-----============= acceptance-small: sanityn ============----- Sun May 17 21:35:28 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg353-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 21:35:49 (1779068149) === Starting client oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Mount client oleg353-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Started clients oleg353-client.virtnet: 192.168.203.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg353-client.virtnet: executing check_config_client /mnt/lustre oleg353-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg353-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c41100ee000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c4110d45800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c41100ee000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c4110d45800.idle_timeout=debug disable quota as required oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:36:28 (1779068188) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463027 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7569] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:36:30 (1779068190) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:36:38 (1779068198) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:36:45 (1779068205) -rw-r--r-- 1 root root 0 May 17 21:36 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:36:54 (1779068214) -rw-r--r-- 1 root root 0 May 17 21:36 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:37:02 (1779068222) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:37:09 (1779068229) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:37:16 (1779068236) total 1 -rwxrwxrwx 1 root root 0 May 17 21:36 f2a -rwxrwxrwx 1 root root 0 May 17 21:36 f2b total 1 -rwxrwxrwx 1 root root 0 May 17 21:36 f2a -rwxrwxrwx 1 root root 0 May 17 21:36 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0162357 s, 31.5 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:37:23 (1779068243) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 21:37:27 (1779068247) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369676 s, 5.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:37:34 (1779068254) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:37:40 (1779068260) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:37:47 (1779068267) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 21:37:52 (1779068272) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:37:59 (1779068279) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:38:05 (1779068285) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:38:13 (1779068293) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 21:38:20 (1779068300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111301 s, 9.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0593102 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:38:28 (1779068308) 1+0 records in 1+0 records out 1 byte copied, 0.0206093 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00798426 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201966 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0220962 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0287766 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0175101 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0321225 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0281902 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0219668 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144661 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0125472 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109765 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:38:38 (1779068318) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0256521 s, 120 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.105368 s, 38.9 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000931178 s, 3.3 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:38:46 (1779068326) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:38:55 (1779068335) warning: '-runtime' deprecated, use '-t runtime' instead - link 299 (time 1779068348.03 total 10.02 last 29.85) - link 596 (time 1779068358.05 total 20.03 last 29.66) - link 936 (time 1779068368.08 total 30.06 last 33.90) - link 1249 (time 1779068378.09 total 40.07 last 31.26) - link 1544 (time 1779068388.12 total 50.10 last 29.43) - link 1844 (time 1779068398.12 total 60.11 last 29.98) - link 2173 (time 1779068408.13 total 70.11 last 32.89) - link 2483 (time 1779068418.13 total 80.11 last 30.99) - link 2787 (time 1779068428.14 total 90.12 last 30.37) total: 3102 link in 100.03 seconds: 31.01 ops/second using seed 2613340041 running for 100 seconds - stat 5316 (time 1779068350 ; total 11 ; last 11) - stat 10000 (time 1779068358 ; total 19 ; last 8) - stat 15346 (time 1779068369 ; total 30 ; last 11) - stat 20000 (time 1779068378 ; total 39 ; last 9) - stat 25339 (time 1779068389 ; total 50 ; last 11) - stat 27940 (time 1779068400 ; total 61 ; last 11) - stat 30000 (time 1779068407 ; total 68 ; last 7) - stat 32786 (time 1779068418 ; total 79 ; last 11) - stat 35614 (time 1779068429 ; total 90 ; last 11) total: 38270 stats in 100 seconds: 382.700012 stats/second - unlinked 0 (time 1779068389 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3102) error: No such file or directory total: 3101 unlinks in 86 seconds: 36.058140 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21373) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21375) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 21:41:20 (1779068480) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:41:29 (1779068489) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22559 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:41:36 (1779068496) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23138 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:41:43 (1779068503) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23716 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:41:49 (1779068509) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24299 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:41:57 (1779068517) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24881 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:42:08 (1779068528) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:43:00 (1779068580) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.510858 s, 16.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.417205 s, 20.1 MB/s Chance of close/open is 1 in 50 Seed set to 8584 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000008[1] 1779068585.072816 mapwrite 0x61ed72 thru 0x62051f (0x17ae bytes) 000012[1] 1779068585.219107 mapread 0x082007 thru 0x0836b3 (0x16ad bytes) 000015[0] 1779068585.337550 mapread 0x3d36bf thru 0x3d5252 (0x1b94 bytes) 000016[0] 1779068585.385678 read 0x4340d8 thru 0x437e4c (0x3d75 bytes) 000017[1] 1779068585.424069 mapwrite 0x4ef211 thru 0x4f7521 (0x8311 bytes) truncating to largest ever: 0x701e09 000018[1] 1779068585.612590 trunc from 0x620520 to 0x701e08 (0xe18e9 bytes) 000019[0] 1779068585.685659 write 0x56f446 thru 0x574182 (0x4d3d bytes) 000023[0] 1779068585.818780 read 0x6a7af8 thru 0x6ab3f2 (0x38fb bytes) 000024[1] 1779068585.847305 write 0x3fed9f thru 0x3ffbab (0xe0d bytes) 000028[0] 1779068585.956761 write 0x1229b6 thru 0x12914c (0x6797 bytes) 000029[1] 1779068586.076777 mapwrite 0x734425 thru 0x73f67f (0xb25b bytes) 000030[0] 1779068586.409737 read 0x4fb6e8 thru 0x508427 (0xcd40 bytes) 000031[1] 1779068586.487269 trunc from 0x73f680 to 0x18284f (0x5bce30 bytes) 000033[0] 1779068586.553445 mapread 0x0e19b8 thru 0x0e9be1 (0x822a bytes) 000034[1] 1779068586.602562 read 0x04cd1e thru 0x058c96 (0xbf79 bytes) 000035[0] 1779068586.636228 write 0x02db7a thru 0x0326ac (0x4b33 bytes) 000036[1] 1779068586.700532 mapread 0x0faae4 thru 0x0fae71 (0x38e bytes) 000037[0] 1779068586.747769 read 0x0b7610 thru 0x0c6b87 (0xf578 bytes) 000038[0] 1779068586.845303 mapread 0x04ecd8 thru 0x05437b (0x56a4 bytes) 000040[0] 1779068586.919522 read 0x11fb15 thru 0x122cb4 (0x31a0 bytes) 000042[1] 1779068586.921641 trunc from 0x182850 to 0x3791f0 (0x1f69a1 bytes) 000043[0] 1779068587.007678 read 0x1b7f5f thru 0x1bedc6 (0x6e68 bytes) 000048[1] 1779068587.310606 write 0x25020f thru 0x25dfbf (0xddb1 bytes) 000050[0] 1779068587.375475 mapread 0x21aa90 thru 0x21d2fa (0x286b bytes) truncating to largest ever: 0x791bd4 000052[0] 1779068587.526007 trunc from 0x3791f1 to 0x791bd3 (0x4189e3 bytes) 000053[1] 1779068587.624117 read 0x53d591 thru 0x541d92 (0x4802 bytes) 000054[1] 1779068587.742013 trunc from 0x791bd4 to 0x28edb3 (0x502e20 bytes) 000055[1] 1779068587.841071 mapread 0x09f226 thru 0x0a6460 (0x723b bytes) 000058[0] 1779068588.082535 write 0x7462a1 thru 0x74b685 (0x53e5 bytes) 000062[1] 1779068588.185374 mapwrite 0x94b6e2 thru 0x951f23 (0x6842 bytes) 000063[1] 1779068588.641651 mapwrite 0x159157 thru 0x16502d (0xbed7 bytes) 000065[1] 1779068588.766302 mapread 0x3bd852 thru 0x3cd3ea (0xfb99 bytes) 000068[0] 1779068588.816331 mapwrite 0x08de3e thru 0x0921c4 (0x4387 bytes) 000071[0] 1779068589.021689 mapwrite 0x5850bf thru 0x589242 (0x4184 bytes) 000072[0] 1779068589.236536 mapwrite 0x83ebf8 thru 0x84bc9a (0xd0a3 bytes) 000073[1] 1779068589.381712 read 0x236af4 thru 0x23bdfa (0x5307 bytes) 000076[1] 1779068589.455605 write 0x6e8993 thru 0x6eb8b8 (0x2f26 bytes) 000077[0] 1779068589.519064 read 0x75d615 thru 0x7653fb (0x7de7 bytes) 000080[1] 1779068589.590027 trunc from 0x951f24 to 0x4475ac (0x50a977 bytes) 000081[1] 1779068589.696614 trunc from 0x4475ad to 0x43cdbc (0xa7f0 bytes) 000085[1] 1779068589.849649 mapread 0x14d48a thru 0x154de8 (0x795f bytes) 000086[0] 1779068589.880685 write 0x63a3a0 thru 0x647cc3 (0xd924 bytes) 000088[0] 1779068589.940822 read 0x305c17 thru 0x311931 (0xbd1b bytes) 000089[0] 1779068589.964640 trunc from 0x647cc4 to 0x2e0381 (0x367942 bytes) 000090[0] 1779068590.019818 read 0x152d32 thru 0x15efef (0xc2be bytes) 000091[1] 1779068590.139501 trunc from 0x2e0382 to 0x53370a (0x253389 bytes) 000096[1] 1779068590.328077 mapwrite 0x4e5f31 thru 0x4e6ff3 (0x10c3 bytes) 000103[0] 1779068590.499636 write 0x6494da thru 0x64c367 (0x2e8e bytes) 000105[1] 1779068590.588647 mapread 0x600c1c thru 0x607161 (0x6546 bytes) 000106[0] 1779068590.657969 mapwrite 0x86c9ea thru 0x871f4a (0x5561 bytes) 000107[0] 1779068591.036288 trunc from 0x871f4b to 0x146f8e (0x72afbc bytes) 000112[0] 1779068591.197280 mapwrite 0x57711d thru 0x5779bc (0x8a0 bytes) 000114[1] 1779068591.269675 mapwrite 0x8fa1e2 thru 0x906384 (0xc1a3 bytes) 000115[0] 1779068591.481458 write 0x3e63ad thru 0x3ec063 (0x5cb7 bytes) 000116[0] 1779068591.554810 read 0x1fa8b1 thru 0x203fef (0x973f bytes) 000119[1] 1779068591.658549 read 0x8b5afb thru 0x8bddfc (0x8302 bytes) 000120[1] 1779068591.726345 mapwrite 0x8ccf21 thru 0x8d7c2a (0xad0a bytes) 000122[1] 1779068591.833405 read 0x786017 thru 0x790364 (0xa34e bytes) truncating to largest ever: 0x868d83 000123[1] 1779068591.856390 trunc from 0x906385 to 0x868d82 (0x9d602 bytes) 000124[0] 1779068591.888070 write 0x3acba3 thru 0x3b5ad6 (0x8f34 bytes) 000125[1] 1779068591.961475 mapread 0x61f604 thru 0x62d890 (0xe28d bytes) 000126[0] 1779068592.008722 mapread 0x7efba5 thru 0x7f24df (0x293b bytes) truncating to largest ever: 0x9ce6ab 000128[1] 1779068592.067544 trunc from 0x868d83 to 0x9ce6aa (0x165928 bytes) 000130[0] 1779068592.170355 mapread 0x37ad8a thru 0x384f67 (0xa1de bytes) 000132[0] 1779068592.230836 mapwrite 0x28b70d thru 0x28c3f3 (0xce7 bytes) 000134[0] 1779068592.385670 mapread 0x214e9c thru 0x21a88f (0x59f4 bytes) 000135[0] 1779068592.495752 read 0x6b693a thru 0x6c0335 (0x99fc bytes) 000136[1] 1779068592.557205 write 0x02be4f thru 0x0394ae (0xd660 bytes) 000137[0] 1779068592.584380 mapread 0x1bec66 thru 0x1ce91b (0xfcb6 bytes) 000138[0] 1779068592.647186 mapwrite 0x38ea39 thru 0x3917df (0x2da7 bytes) 000140[1] 1779068592.747282 mapwrite 0x669b14 thru 0x669c90 (0x17d bytes) 000144[1] 1779068592.977248 trunc from 0x9ce6ab to 0x3f8e4a (0x5d5860 bytes) 000147[1] 1779068593.185942 mapread 0x3931ee thru 0x397700 (0x4513 bytes) 000148[0] 1779068593.231858 write 0x6862eb thru 0x689b84 (0x389a bytes) 000150[0] 1779068593.313764 read 0x565569 thru 0x56cfce (0x7a66 bytes) 000152[1] 1779068593.365957 mapread 0x57a75a thru 0x587bf4 (0xd49b bytes) 000153[0] 1779068593.443402 read 0x195483 thru 0x195741 (0x2bf bytes) 000154[0] 1779068593.473347 mapwrite 0x442edf thru 0x44e479 (0xb59b bytes) 000156[1] 1779068593.597214 trunc from 0x689b85 to 0x45c1ba (0x22d9ca bytes) 000159[1] 1779068593.738125 mapwrite 0x2aab16 thru 0x2adaf1 (0x2fdc bytes) 000166[1] 1779068594.096855 mapread 0x1b75af thru 0x1b88c2 (0x1314 bytes) 000170[1] 1779068594.270362 mapwrite 0x1892e3 thru 0x19905d (0xfd7b bytes) 000171[1] 1779068594.391509 read 0x3fe31d thru 0x4093d5 (0xb0b9 bytes) 000173[0] 1779068594.470263 read 0x32c1eb thru 0x330b68 (0x497e bytes) 000174[1] 1779068594.503421 read 0x05e455 thru 0x05ffe1 (0x1b8d bytes) 000177[0] 1779068594.526694 write 0x040391 thru 0x04e499 (0xe109 bytes) 000178[1] 1779068594.576784 read 0x3e990b thru 0x3f9487 (0xfb7d bytes) 000179[1] 1779068594.633279 write 0x6976ee thru 0x69be42 (0x4755 bytes) 000180[1] 1779068594.638924 write 0x641c84 thru 0x644f12 (0x328f bytes) 000181[0] 1779068594.667397 write 0x7ad341 thru 0x7b47f6 (0x74b6 bytes) 000183[1] 1779068594.716537 mapread 0x0a56c3 thru 0x0b44ba (0xedf8 bytes) 000184[1] 1779068594.758325 mapwrite 0x719bc3 thru 0x72352d (0x996b bytes) 000186[0] 1779068594.929581 mapread 0x4a8ca3 thru 0x4b04bf (0x781d bytes) 000187[1] 1779068595.010034 mapread 0x4d0a2c thru 0x4d0d4f (0x324 bytes) 000189[0] 1779068595.022988 read 0x283528 thru 0x2918c8 (0xe3a1 bytes) 000192[0] 1779068595.078339 trunc from 0x7b47f7 to 0x0eb39a (0x6c945c bytes) 000193[1] 1779068595.263590 write 0x971bd7 thru 0x97d645 (0xba6f bytes) 000194[1] 1779068595.363422 mapread 0x40733d thru 0x40c6d6 (0x539a bytes) 000195[0] 1779068595.426171 trunc from 0x97d646 to 0x08e48d (0x8ef1b8 bytes) 000196[1] 1779068595.485460 trunc from 0x08e48e to 0x1af4e9 (0x12105c bytes) 000197[0] 1779068595.537551 write 0x23ba77 thru 0x23d8f1 (0x1e7b bytes) 000201[1] 1779068595.639265 read 0x19364e thru 0x1a30bd (0xfa70 bytes) 000203[1] 1779068595.749454 trunc from 0x23d8f2 to 0x36ad34 (0x12d443 bytes) 000204[1] 1779068595.797284 mapwrite 0x61fd06 thru 0x624839 (0x4b34 bytes) 000207[0] 1779068596.081372 mapwrite 0x94deba thru 0x951e22 (0x3f69 bytes) 000209[1] 1779068596.319158 mapread 0x5e4a2a thru 0x5ef6b0 (0xac87 bytes) 000211[0] 1779068596.419266 write 0x5ec7a8 thru 0x5f132b (0x4b84 bytes) 000213[0] 1779068596.508204 read 0x334af3 thru 0x33587a (0xd88 bytes) 000214[0] 1779068596.555817 trunc from 0x951e23 to 0x777234 (0x1dabee bytes) 000215[1] 1779068596.634469 trunc from 0x777235 to 0x5df452 (0x197de2 bytes) 000216[1] 1779068596.748499 read 0x34292f thru 0x34298c (0x5e bytes) 000218[0] 1779068596.829207 mapread 0x2f84a8 thru 0x304314 (0xbe6d bytes) 000227[1] 1779068597.079638 trunc from 0x5df453 to 0x103fbc (0x4db496 bytes) 000228[1] 1779068597.133274 trunc from 0x103fbd to 0x5c4dc3 (0x4c0e07 bytes) 000229[0] 1779068597.194598 mapwrite 0x07fee7 thru 0x0892bb (0x93d5 bytes) 000230[0] 1779068597.334650 trunc from 0x5c4dc4 to 0x98107b (0x3bc2b8 bytes) 000231[0] 1779068597.426081 trunc from 0x98107c to 0x739604 (0x247a77 bytes) 000232[1] 1779068597.456958 trunc from 0x739605 to 0x05e13d (0x6db4c7 bytes) 000233[0] 1779068597.604773 read 0x0503ef thru 0x05e13d (0xdd4f bytes) 000234[0] 1779068597.671308 mapread 0x021da1 thru 0x031639 (0xf899 bytes) 000235[1] 1779068597.724032 mapwrite 0x22992c thru 0x236431 (0xcb06 bytes) 000236[0] 1779068598.022527 mapread 0x034cd1 thru 0x043dd7 (0xf107 bytes) 000237[0] 1779068598.288190 mapwrite 0x0bb754 thru 0x0bef93 (0x3840 bytes) 000244[1] 1779068598.700547 trunc from 0x236432 to 0x9a1331 (0x76af00 bytes) 000246[0] 1779068598.836981 read 0x8c77c0 thru 0x8d56b7 (0xdef8 bytes) 000249[0] 1779068598.920129 trunc from 0x9a1332 to 0x50c558 (0x494dd9 bytes) 000250[0] 1779068599.015461 read 0x3437f6 thru 0x34c3f8 (0x8c03 bytes) 000251[1] 1779068599.028845 read 0x17a706 thru 0x17c7fb (0x20f6 bytes) 000252[1] 1779068599.087982 mapwrite 0x277ac2 thru 0x285ff7 (0xe536 bytes) 000253[1] 1779068599.271717 read 0x1e4eba thru 0x1ee2d8 (0x941f bytes) 000256[0] 1779068599.352999 write 0x983e3b thru 0x98e756 (0xa91c bytes) 000258[1] 1779068599.414358 mapwrite 0x972801 thru 0x980e40 (0xe640 bytes) 000259[0] 1779068599.680017 trunc from 0x98e757 to 0x607f4d (0x386809 bytes) 000260[0] 1779068599.769032 mapwrite 0x33e23c thru 0x34846b (0xa230 bytes) 000263[1] 1779068600.069455 mapread 0x0ebf71 thru 0x0f1131 (0x51c1 bytes) 000265[1] 1779068600.200779 read 0x30970f thru 0x316af6 (0xd3e8 bytes) 000266[1] 1779068600.219596 read 0x3eae43 thru 0x3f29fe (0x7bbc bytes) 000268[0] 1779068600.239117 mapread 0x604284 thru 0x607f4d (0x3cca bytes) 000271[0] 1779068600.288271 write 0x819d93 thru 0x827b94 (0xde02 bytes) 000273[0] 1779068600.361788 mapread 0x13e019 thru 0x146609 (0x85f1 bytes) 000276[1] 1779068600.409618 trunc from 0x827b95 to 0x5c1284 (0x266910 bytes) 000277[0] 1779068600.460295 read 0x4daaa4 thru 0x4dcdb5 (0x2312 bytes) 000280[0] 1779068600.501778 mapwrite 0x1c732c thru 0x1c8937 (0x160c bytes) 000284[1] 1779068600.659679 mapwrite 0x732d91 thru 0x73ec9a (0xbf0a bytes) 000285[1] 1779068600.901405 read 0x3f4c83 thru 0x404995 (0xfd13 bytes) 000287[1] 1779068600.981269 mapwrite 0x774837 thru 0x781187 (0xc951 bytes) 000288[1] 1779068601.146289 mapwrite 0x559505 thru 0x56475c (0xb258 bytes) 000289[0] 1779068601.232675 write 0x36ae9d thru 0x370a0b (0x5b6f bytes) 000291[0] 1779068601.291380 trunc from 0x781188 to 0x8248c4 (0xa373d bytes) 000292[0] 1779068601.347979 write 0x65d4aa thru 0x668275 (0xadcc bytes) 000293[1] 1779068601.424078 trunc from 0x8248c5 to 0x46bc75 (0x3b8c4f bytes) 000295[0] 1779068601.555207 mapread 0x3d1568 thru 0x3df936 (0xe3cf bytes) 000298[1] 1779068601.629631 write 0x1b5119 thru 0x1c35e9 (0xe4d1 bytes) 000299[1] 1779068601.676678 write 0x8f1e31 thru 0x8fc3e1 (0xa5b1 bytes) 000302[0] 1779068601.785974 trunc from 0x8fc3e2 to 0x788f3f (0x1734a2 bytes) 000304[0] 1779068601.924811 mapwrite 0x14d304 thru 0x158f74 (0xbc71 bytes) 000306[0] 1779068602.067041 mapread 0x6179e7 thru 0x61a562 (0x2b7c bytes) 000310[1] 1779068602.114700 write 0x87e589 thru 0x884812 (0x628a bytes) 000311[1] 1779068602.145092 trunc from 0x884813 to 0x3d5ebc (0x4ae956 bytes) 000313[0] 1779068602.237241 mapwrite 0x31f515 thru 0x32ad8e (0xb87a bytes) 000317[0] 1779068602.431453 mapwrite 0x20be06 thru 0x2155f6 (0x97f1 bytes) 000320[1] 1779068602.547628 write 0x75af3f thru 0x7603da (0x549c bytes) 000325[0] 1779068602.629705 mapwrite 0x2abad2 thru 0x2b5e08 (0xa337 bytes) 000326[0] 1779068602.762982 trunc from 0x7603db to 0x3175b6 (0x448e24 bytes) 000327[0] 1779068602.857260 mapread 0x28353e thru 0x2837c1 (0x284 bytes) 000329[1] 1779068602.881783 write 0x2d183a thru 0x2e0ad6 (0xf29d bytes) 000330[1] 1779068602.934748 read 0x025589 thru 0x0302cd (0xad45 bytes) 000332[1] 1779068603.073490 read 0x020bfc thru 0x027a23 (0x6e28 bytes) 000334[1] 1779068603.144175 read 0x100332 thru 0x103b03 (0x37d2 bytes) 000335[0] 1779068603.163455 read 0x095d8d thru 0x0a079e (0xaa12 bytes) 000337[1] 1779068603.276636 mapwrite 0x9dc0ee thru 0x9ddede (0x1df1 bytes) 000341[0] 1779068603.683257 read 0x376723 thru 0x381fb3 (0xb891 bytes) 000343[0] 1779068603.717832 mapwrite 0x7c92ff thru 0x7d3c3d (0xa93f bytes) 000344[1] 1779068603.878646 mapread 0x5f6a35 thru 0x5f7928 (0xef4 bytes) 000345[1] 1779068603.936042 trunc from 0x9ddedf to 0x407048 (0x5d6e96 bytes) 000347[0] 1779068604.050326 read 0x27d823 thru 0x27f8ea (0x20c8 bytes) 000351[0] 1779068604.163535 trunc from 0x407049 to 0x12c5d4 (0x2daa74 bytes) 000353[1] 1779068604.327340 trunc from 0x12c5d5 to 0x7c3e40 (0x69786c bytes) 000354[1] 1779068604.429407 mapread 0x46cc37 thru 0x470d95 (0x415f bytes) 000359[0] 1779068604.593572 mapread 0x436e92 thru 0x445437 (0xe5a6 bytes) 000361[1] 1779068604.679371 read 0x1d6348 thru 0x1d7890 (0x1549 bytes) 000368[0] 1779068604.816323 mapread 0x041da1 thru 0x04ae3f (0x909f bytes) 000370[1] 1779068604.856948 trunc from 0x7c3e41 to 0x95be02 (0x197fc2 bytes) 000371[0] 1779068604.953500 read 0x0bd3fa thru 0x0c2e93 (0x5a9a bytes) 000372[0] 1779068604.995523 read 0x24871a thru 0x24ef99 (0x6880 bytes) 000373[1] 1779068605.016679 trunc from 0x95be03 to 0x9c6642 (0x6a840 bytes) 000375[0] 1779068605.102667 mapwrite 0x28cac7 thru 0x2993e7 (0xc921 bytes) 000376[1] 1779068605.232192 read 0x7bd1d2 thru 0x7c9d11 (0xcb40 bytes) 000378[1] 1779068605.285016 write 0x73de93 thru 0x743b50 (0x5cbe bytes) 000379[1] 1779068605.346161 trunc from 0x9c6643 to 0x97f582 (0x470c0 bytes) 000381[0] 1779068605.477755 mapwrite 0x44804f thru 0x45744c (0xf3fe bytes) 000382[0] 1779068605.644680 write 0x564d33 thru 0x56a870 (0x5b3e bytes) 000384[0] 1779068605.687141 read 0x59a241 thru 0x5a9683 (0xf443 bytes) 000388[1] 1779068605.767245 mapread 0x2abb4b thru 0x2b2093 (0x6549 bytes) 000390[1] 1779068605.836274 mapread 0x805257 thru 0x806062 (0xe0c bytes) 000391[1] 1779068605.872067 read 0x754025 thru 0x760dd2 (0xcdae bytes) 000394[1] 1779068605.940206 trunc from 0x97f583 to 0x3be71d (0x5c0e65 bytes) 000397[1] 1779068606.010330 trunc from 0x3be71e to 0x19d44c (0x2212d1 bytes) 000398[0] 1779068606.064592 mapwrite 0x5fb236 thru 0x60a62c (0xf3f7 bytes) 000399[0] 1779068606.252802 trunc from 0x60a62d to 0x262fc5 (0x3a7667 bytes) 000400[1] 1779068606.278838 write 0x35792e thru 0x363593 (0xbc66 bytes) 000402[1] 1779068606.390760 write 0x68b444 thru 0x694af9 (0x96b6 bytes) 000403[0] 1779068606.447217 read 0x06eaf7 thru 0x07d7c8 (0xecd2 bytes) 000404[0] 1779068606.497744 read 0x5d1ef7 thru 0x5d8462 (0x656c bytes) 000414[0] 1779068606.537302 mapwrite 0x067fa7 thru 0x06f2ea (0x7344 bytes) 000415[0] 1779068606.667957 read 0x04100b thru 0x047197 (0x618d bytes) 000421[1] 1779068606.711969 read 0x5f23fa thru 0x5f9a8c (0x7693 bytes) 000422[1] 1779068606.737854 mapread 0x52a62b thru 0x5378d1 (0xd2a7 bytes) 000423[1] 1779068606.773583 mapread 0x572c38 thru 0x57fd0c (0xd0d5 bytes) 000427[0] 1779068606.837683 trunc from 0x694afa to 0x10b278 (0x589881 bytes) 000428[1] 1779068606.901486 read 0x0ff9da thru 0x10567f (0x5ca6 bytes) 000430[1] 1779068606.961209 trunc from 0x10b279 to 0x4fd028 (0x3f1db0 bytes) 000435[1] 1779068607.079596 mapread 0x1d9f21 thru 0x1e8c23 (0xed03 bytes) 000436[0] 1779068607.170233 write 0x597c43 thru 0x597fa7 (0x365 bytes) 000438[0] 1779068607.220821 mapread 0x1fb1f9 thru 0x1ff5d3 (0x43db bytes) 000444[1] 1779068607.367185 mapread 0x59094e thru 0x597fa7 (0x765a bytes) 000449[0] 1779068607.439865 trunc from 0x597fa8 to 0x771590 (0x1d95e9 bytes) 000450[1] 1779068607.502646 mapwrite 0x22e567 thru 0x230f4e (0x29e8 bytes) 000453[1] 1779068607.620125 trunc from 0x771591 to 0x84ec10 (0xdd680 bytes) 000458[1] 1779068607.730486 trunc from 0x84ec11 to 0x2a703c (0x5a7bd4 bytes) 000459[0] 1779068607.775848 mapread 0x288810 thru 0x29788d (0xf07e bytes) 000460[1] 1779068607.829700 mapread 0x1db5eb thru 0x1e5e34 (0xa84a bytes) 000461[0] 1779068607.922600 write 0x1de064 thru 0x1e281a (0x47b7 bytes) 000462[1] 1779068607.958762 read 0x08e741 thru 0x095b86 (0x7446 bytes) 000463[0] 1779068608.068769 trunc from 0x2a703d to 0x4c4340 (0x21d304 bytes) 000467[1] 1779068608.273097 mapread 0x1b2ecc thru 0x1b3fa7 (0x10dc bytes) 000470[1] 1779068608.386406 mapwrite 0x756b08 thru 0x75adf8 (0x42f1 bytes) 000474[0] 1779068608.652209 mapread 0x5f5759 thru 0x6045b4 (0xee5c bytes) 000475[0] 1779068608.729226 read 0x5b42a4 thru 0x5b71c7 (0x2f24 bytes) 000479[0] 1779068608.752337 mapwrite 0x7ed083 thru 0x7fa194 (0xd112 bytes) 000480[0] 1779068608.875064 write 0x523118 thru 0x530fd4 (0xdebd bytes) 000481[0] 1779068608.935397 mapread 0x0e3bca thru 0x0e6518 (0x294f bytes) 000482[1] 1779068608.997504 read 0x05d9c6 thru 0x061116 (0x3751 bytes) 000484[1] 1779068609.058658 mapwrite 0x9379e6 thru 0x93fc09 (0x8224 bytes) 000485[0] 1779068609.226178 mapread 0x2d62f3 thru 0x2dd3c3 (0x70d1 bytes) 000489[0] 1779068609.411986 read 0x26bab1 thru 0x27197a (0x5eca bytes) 000491[0] 1779068609.463924 read 0x087e44 thru 0x08cdb8 (0x4f75 bytes) 000498[0] 1779068609.600200 write 0x7ec76b thru 0x7eee66 (0x26fc bytes) 000499[1] 1779068609.629774 mapread 0x051bc9 thru 0x056665 (0x4a9d bytes) 000500[1] 1779068609.665734 write 0x90f5d7 thru 0x9111a6 (0x1bd0 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8609 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x82d49b 000012[0] 1779068611.040853 trunc from 00000000 to 0x82d49a (0x82d49b bytes) 000013[1] 1779068611.130342 mapwrite 0x48c000 thru 0x497f42 (0xbf43 bytes) 000014[0] 1779068611.396570 read 0x4c9000 thru 0x4d5b75 (0xcb76 bytes) 000016[1] 1779068611.426393 mapread 0x017000 thru 0x017708 (0x709 bytes) 000019[1] 1779068611.432648 read_OD 0x617000 thru 0x625fff (0xf000 bytes) 000021[1] 1779068611.441414 mapwrite 0x424000 thru 0x430895 (0xc896 bytes) 000029[0] 1779068611.627274 trunc from 0x82d49b to 0x21c9a2 (0x610af8 bytes) 000034[0] 1779068611.740815 trunc from 0x21c9a3 to 0x3653a1 (0x1489ff bytes) 000035[0] 1779068611.774388 read 0x1c4000 thru 0x1cddc6 (0x9dc7 bytes) 000037[0] 1779068611.873700 mapwrite 0x7f4000 thru 0x7f9cf5 (0x5cf6 bytes) 000040[1] 1779068612.114193 write_OD 0x3cd000 thru 0x3dafff (0xe000 bytes) 000043[0] 1779068612.215644 mapwrite 0x4cf000 thru 0x4dbf74 (0xcf75 bytes) 000044[1] 1779068612.433752 mapwrite 0x2ea000 thru 0x2ed4c5 (0x34c6 bytes) 000046[1] 1779068612.543939 mapread 0x7b9000 thru 0x7bf55e (0x655f bytes) 000048[1] 1779068612.584649 read_OD 0x4ac000 thru 0x4b8fff (0xd000 bytes) 000051[0] 1779068612.630133 trunc from 0x7f9cf6 to 0x3fb3e7 (0x3fe90e bytes) 000052[1] 1779068612.684991 mapread 0x3c8000 thru 0x3d5b65 (0xdb66 bytes) 000055[0] 1779068612.744395 read 0x149000 thru 0x15281e (0x981f bytes) 000057[0] 1779068612.837969 mapwrite 0x7fb000 thru 0x7fd920 (0x2921 bytes) 000059[0] 1779068613.112829 mapread 0x77c000 thru 0x77e349 (0x234a bytes) 000060[1] 1779068613.113892 mapread 0x60d000 thru 0x60f75e (0x275f bytes) 000061[1] 1779068613.193961 read_OD 0x099000 thru 0x0a6fff (0xe000 bytes) 000063[0] 1779068613.218615 mapread 0x4be000 thru 0x4c5e04 (0x7e05 bytes) 000064[1] 1779068613.265180 read_OD 0x480000 thru 0x48afff (0xb000 bytes) 000065[0] 1779068613.274644 trunc from 0x7fd921 to 0x69f914 (0x15e00c bytes) 000066[1] 1779068613.321937 mapread 0x597000 thru 0x5a0ecf (0x9ed0 bytes) 000067[0] 1779068613.378645 write 0x752000 thru 0x7547fe (0x27ff bytes) 000068[0] 1779068613.397608 mapread 0x4de000 thru 0x4e0770 (0x2771 bytes) 000072[1] 1779068613.486612 read_OD 0x355000 thru 0x35afff (0x6000 bytes) 000078[1] 1779068613.567982 mapwrite 0x164000 thru 0x16eb8a (0xab8b bytes) truncating to largest ever: 0x95446e 000079[1] 1779068613.684085 trunc from 0x7547ff to 0x95446d (0x1ffc6f bytes) 000081[0] 1779068613.712493 trunc from 0x95446e to 0x85af2e (0xf953f bytes) 000082[0] 1779068613.748450 trunc from 0x85af2f to 0x12a8b8 (0x730676 bytes) 000083[0] 1779068613.766536 write 0x719000 thru 0x71bdae (0x2daf bytes) 000084[1] 1779068613.771377 mapwrite 0x5c0000 thru 0x5c4974 (0x4975 bytes) 000087[1] 1779068613.935816 read_OD 0x39a000 thru 0x39dfff (0x4000 bytes) 000093[0] 1779068613.960303 mapwrite 0x2da000 thru 0x2e1db7 (0x7db8 bytes) 000095[1] 1779068614.083356 read_OD 0x373000 thru 0x380fff (0xe000 bytes) 000098[0] 1779068614.163238 trunc from 0x71bdaf to 0x2a15fa (0x47a7b4 bytes) 000099[0] 1779068614.228424 write 0x774000 thru 0x77aa70 (0x6a71 bytes) 000100[1] 1779068614.237219 mapwrite 0x7cd000 thru 0x7d8a94 (0xba95 bytes) 000107[1] 1779068614.483433 mapwrite 0x066000 thru 0x0708dc (0xa8dd bytes) 000108[1] 1779068614.579137 trunc from 0x7d8a95 to 0x30c9f1 (0x4cc0a3 bytes) 000109[1] 1779068614.604591 mapread 0x213000 thru 0x222793 (0xf794 bytes) 000111[1] 1779068614.676243 mapread 0x0e8000 thru 0x0edf9a (0x5f9b bytes) 000112[1] 1779068614.722707 write_OD 0x457000 thru 0x458fff (0x2000 bytes) 000113[1] 1779068614.819632 mapwrite 0x614000 thru 0x61624a (0x224b bytes) 000114[1] 1779068614.973643 write_OD 0x96b000 thru 0x970fff (0x6000 bytes) 000116[0] 1779068615.095997 read 0x500000 thru 0x50e393 (0xe394 bytes) truncating to largest ever: 0x968c3f 000121[1] 1779068615.136675 trunc from 0x971000 to 0x968c3e (0x83c1 bytes) 000122[0] 1779068615.235278 read 0x727000 thru 0x736c5d (0xfc5e bytes) 000123[1] 1779068615.270713 read_OD 0x549000 thru 0x549fff (0x1000 bytes) 000130[0] 1779068615.299234 mapwrite 0x300000 thru 0x30de49 (0xde4a bytes) 000131[0] 1779068615.410473 mapwrite 0x62f000 thru 0x630f0a (0x1f0b bytes) 000134[0] 1779068615.562861 mapread 0x21b000 thru 0x228cbe (0xdcbf bytes) 000135[0] 1779068615.615904 mapread 0x521000 thru 0x52aec7 (0x9ec8 bytes) 000138[1] 1779068615.690648 mapread 0x66d000 thru 0x67b16c (0xe16d bytes) 000141[0] 1779068615.755688 mapwrite 0x339000 thru 0x346504 (0xd505 bytes) 000142[1] 1779068615.922723 mapwrite 0x86e000 thru 0x872c52 (0x4c53 bytes) 000143[0] 1779068616.101160 mapread 0x5d2000 thru 0x5dee8e (0xce8f bytes) 000144[1] 1779068616.169687 mapread 0x160000 thru 0x167a12 (0x7a13 bytes) 000146[0] 1779068616.192334 write 0x9ec000 thru 0x9f00fb (0x40fc bytes) 000148[0] 1779068616.240554 read 0x1df000 thru 0x1e99d4 (0xa9d5 bytes) 000149[1] 1779068616.253983 mapread 0x2f0000 thru 0x2f93e9 (0x93ea bytes) 000150[0] 1779068616.305250 mapread 0x135000 thru 0x13a47a (0x547b bytes) 000154[1] 1779068616.326650 mapwrite 0x2bc000 thru 0x2bd2ff (0x1300 bytes) 000155[1] 1779068616.423169 trunc from 0x9f00fc to 0x7e809b (0x208060 bytes) 000156[1] 1779068616.448304 read_OD 0x502000 thru 0x50ffff (0xe000 bytes) 000159[0] 1779068616.483798 mapwrite 0x4a4000 thru 0x4ae9d6 (0xa9d7 bytes) 000160[1] 1779068616.608757 mapread 0x3ff000 thru 0x40e57f (0xf580 bytes) 000163[1] 1779068616.679191 trunc from 0x7e809c to 0x8c58d4 (0xdd839 bytes) 000165[0] 1779068616.786467 read 0x731000 thru 0x73af6e (0x9f6f bytes) 000166[0] 1779068616.811850 mapwrite 0x489000 thru 0x4935c8 (0xa5c9 bytes) 000168[0] 1779068616.980698 read 0x1ba000 thru 0x1c3b39 (0x9b3a bytes) 000169[0] 1779068617.009637 mapread 0x4db000 thru 0x4e278f (0x7790 bytes) 000172[0] 1779068617.054751 trunc from 0x8c58d5 to 0x0e9098 (0x7dc83c bytes) 000177[1] 1779068617.226900 write_OD 0x31e000 thru 0x31efff (0x1000 bytes) 000178[1] 1779068617.299090 read_OD 0x0b9000 thru 0x0bafff (0x2000 bytes) 000181[1] 1779068617.311112 write_OD 0x2a1000 thru 0x2a6fff (0x6000 bytes) 000194[1] 1779068617.370428 read_OD 0x02f000 thru 0x038fff (0xa000 bytes) 000196[1] 1779068617.377791 mapwrite 0x84a000 thru 0x84e37a (0x437b bytes) 000197[0] 1779068617.799751 trunc from 0x84e37b to 0x6e8764 (0x165c16 bytes) 000199[1] 1779068617.867307 mapwrite 0x434000 thru 0x43724a (0x324b bytes) 000200[1] 1779068618.049930 write_OD 0x91e000 thru 0x920fff (0x3000 bytes) 000202[0] 1779068618.209344 mapwrite 0x03f000 thru 0x04e283 (0xf284 bytes) 000204[1] 1779068618.457484 mapread 0x477000 thru 0x47cca0 (0x5ca1 bytes) 000208[1] 1779068618.543356 mapread 0x00f000 thru 0x013572 (0x4573 bytes) 000209[1] 1779068618.572361 mapwrite 0x340000 thru 0x347890 (0x7891 bytes) 000210[1] 1779068618.830955 mapwrite 0x32d000 thru 0x32fce9 (0x2cea bytes) 000211[0] 1779068618.891087 read 0x6a1000 thru 0x6a9938 (0x8939 bytes) 000213[0] 1779068618.917218 mapread 0x311000 thru 0x31c72b (0xb72c bytes) 000216[0] 1779068618.973465 mapread 0x31d000 thru 0x32c62e (0xf62f bytes) 000219[1] 1779068618.987561 write_OD 0x508000 thru 0x50bfff (0x4000 bytes) 000221[1] 1779068619.090369 mapwrite 0x80f000 thru 0x811af9 (0x2afa bytes) 000222[1] 1779068619.179734 write_OD 0x7d6000 thru 0x7dcfff (0x7000 bytes) 000223[1] 1779068619.245755 trunc from 0x921000 to 0x0df002 (0x841ffd bytes) 000225[0] 1779068619.320326 read 0x044000 thru 0x0496b2 (0x56b3 bytes) 000226[0] 1779068619.377280 mapwrite 0x903000 thru 0x911a64 (0xea65 bytes) 000227[1] 1779068619.661495 read_OD 0x8a4000 thru 0x8abfff (0x8000 bytes) 000228[0] 1779068619.710410 mapwrite 0x94e000 thru 0x955c4e (0x7c4f bytes) 000231[0] 1779068619.823292 read 0x0db000 thru 0x0de49e (0x349f bytes) 000232[0] 1779068619.848129 trunc from 0x955c4f to 0x31c950 (0x6392fe bytes) 000233[1] 1779068619.883567 write_OD 0x63c000 thru 0x641fff (0x6000 bytes) 000240[0] 1779068619.997069 mapread 0x170000 thru 0x17092e (0x92f bytes) 000241[0] 1779068620.008457 trunc from 0x642000 to 0x4bb667 (0x186998 bytes) 000242[1] 1779068620.110145 mapwrite 0x9cb000 thru 0x9cb71e (0x71f bytes) 000244[0] 1779068620.319267 read 0x9b7000 thru 0x9bae5d (0x3e5e bytes) 000247[0] 1779068620.361392 read 0x8c2000 thru 0x8cde8f (0xbe90 bytes) 000249[0] 1779068620.372846 mapwrite 0x19d000 thru 0x1aa2c5 (0xd2c6 bytes) 000251[0] 1779068620.516152 trunc from 0x9cb71f to 0x58c82c (0x43eef2 bytes) 000253[0] 1779068620.560563 mapread 0x41c000 thru 0x4221f8 (0x61f9 bytes) 000255[1] 1779068620.593527 trunc from 0x58c82d to 0x5e6de9 (0x5a5bd bytes) 000258[1] 1779068620.652593 trunc from 0x5e6dea to 0x7a0352 (0x1b9569 bytes) 000259[0] 1779068620.691296 read 0x6f5000 thru 0x702275 (0xd276 bytes) 000262[0] 1779068620.715820 mapwrite 0x760000 thru 0x76042f (0x430 bytes) 000264[0] 1779068620.876636 mapread 0x22b000 thru 0x23a471 (0xf472 bytes) 000265[0] 1779068620.968480 mapread 0x2a9000 thru 0x2b54dd (0xc4de bytes) 000266[0] 1779068621.056028 write 0x590000 thru 0x59d4a1 (0xd4a2 bytes) truncating to largest ever: 0x99ec5f 000270[0] 1779068621.130573 trunc from 0x7a0353 to 0x99ec5e (0x1fe90c bytes) 000271[1] 1779068621.158680 read_OD 0x727000 thru 0x72bfff (0x5000 bytes) 000273[1] 1779068621.233503 trunc from 0x99ec5f to 0x864387 (0x13a8d7 bytes) 000274[1] 1779068621.292543 mapwrite 0x310000 thru 0x31f7a1 (0xf7a2 bytes) 000275[0] 1779068621.397236 mapwrite 0x7a5000 thru 0x7b4b01 (0xfb02 bytes) 000277[1] 1779068621.519179 mapread 0x0d5000 thru 0x0dc46f (0x7470 bytes) 000279[1] 1779068621.565732 mapread 0x5c9000 thru 0x5d47c6 (0xb7c7 bytes) 000281[1] 1779068621.617509 write_OD 0x5e4000 thru 0x5e9fff (0x6000 bytes) 000283[1] 1779068621.725455 mapread 0x372000 thru 0x3773ec (0x53ed bytes) 000286[0] 1779068621.776798 write 0x589000 thru 0x58efc2 (0x5fc3 bytes) 000287[0] 1779068621.813199 read 0x541000 thru 0x54f1be (0xe1bf bytes) 000288[0] 1779068621.842663 mapwrite 0x2b8000 thru 0x2b8239 (0x23a bytes) 000289[1] 1779068621.933247 read_OD 0x57e000 thru 0x585fff (0x8000 bytes) 000291[0] 1779068621.973588 mapread 0x4ab000 thru 0x4b5ace (0xaacf bytes) 000294[0] 1779068622.021136 trunc from 0x864388 to 0x306376 (0x55e011 bytes) 000298[0] 1779068622.084540 mapwrite 0x78c000 thru 0x79121e (0x521f bytes) 000300[0] 1779068622.195422 trunc from 0x79121f to 0x21aad9 (0x576745 bytes) 000301[0] 1779068622.232345 mapwrite 0x9db000 thru 0x9e8f05 (0xdf06 bytes) 000303[0] 1779068622.368489 mapread 0x8e9000 thru 0x8f5187 (0xc188 bytes) 000305[1] 1779068622.424765 trunc from 0x9e8f06 to 0x959537 (0x8f9ce bytes) 000306[1] 1779068622.458892 write_OD 0x265000 thru 0x272fff (0xe000 bytes) 000307[0] 1779068622.596723 mapwrite 0x23a000 thru 0x245d1a (0xbd1b bytes) 000309[1] 1779068622.860983 read_OD 0x22b000 thru 0x236fff (0xc000 bytes) 000310[1] 1779068622.915323 mapwrite 0x226000 thru 0x231c52 (0xbc53 bytes) 000311[0] 1779068623.051745 mapwrite 0x622000 thru 0x626ad5 (0x4ad6 bytes) 000312[1] 1779068623.157258 read_OD 0x19d000 thru 0x1a1fff (0x5000 bytes) 000314[1] 1779068623.185148 mapread 0x3ae000 thru 0x3b9993 (0xb994 bytes) 000320[1] 1779068623.215177 mapread 0x559000 thru 0x5630b1 (0xa0b2 bytes) 000325[0] 1779068623.235329 read 0x759000 thru 0x761ffe (0x8fff bytes) 000328[1] 1779068623.245352 read_OD 0x202000 thru 0x20bfff (0xa000 bytes) 000331[1] 1779068623.260231 trunc from 0x959538 to 0x70fc86 (0x2498b1 bytes) 000332[1] 1779068623.315411 trunc from 0x70fc87 to 0x66af62 (0xa4d24 bytes) 000333[1] 1779068623.393240 read_OD 0x61e000 thru 0x622fff (0x5000 bytes) 000335[0] 1779068623.471494 mapread 0x2c6000 thru 0x2d577f (0xf780 bytes) 000336[0] 1779068623.561721 mapwrite 0x1fa000 thru 0x1fdd8b (0x3d8c bytes) 000341[0] 1779068623.768511 mapread 0x5ca000 thru 0x5d37b2 (0x97b3 bytes) 000344[0] 1779068623.860782 write 0x156000 thru 0x158d8d (0x2d8e bytes) 000346[0] 1779068623.894605 mapwrite 0x92c000 thru 0x938ccf (0xccd0 bytes) 000347[1] 1779068624.015813 trunc from 0x938cd0 to 0x81bdbb (0x11cf14 bytes) 000348[1] 1779068624.077564 mapwrite 0x44d000 thru 0x456c59 (0x9c5a bytes) 000351[1] 1779068624.189993 read_OD 0x66c000 thru 0x676fff (0xb000 bytes) 000353[0] 1779068624.220416 mapwrite 0x318000 thru 0x322e7e (0xae7f bytes) 000355[1] 1779068624.355098 mapwrite 0x946000 thru 0x94cf5f (0x6f60 bytes) 000357[0] 1779068624.571634 read 0x7c6000 thru 0x7caaab (0x4aac bytes) 000361[0] 1779068624.628976 read 0x00e000 thru 0x013e71 (0x5e72 bytes) 000363[0] 1779068624.641784 mapwrite 0x4b9000 thru 0x4c7a66 (0xea67 bytes) 000366[0] 1779068624.859227 read 0x10f000 thru 0x11d29b (0xe29c bytes) 000367[0] 1779068624.875156 trunc from 0x94cf60 to 0x511efd (0x43b062 bytes) 000368[1] 1779068624.908288 mapread 0x4a0000 thru 0x4a4bf0 (0x4bf1 bytes) 000369[1] 1779068624.963147 read_OD 0x017000 thru 0x021fff (0xb000 bytes) 000370[0] 1779068624.986960 trunc from 0x511efe to 0x36d16b (0x1a4d92 bytes) 000371[0] 1779068625.042956 trunc from 0x36d16c to 0x4b81fa (0x14b08f bytes) 000372[1] 1779068625.079931 mapwrite 0x1da000 thru 0x1e72bd (0xd2be bytes) 000374[0] 1779068625.292835 write 0x84c000 thru 0x85b793 (0xf794 bytes) 000375[1] 1779068625.333862 read_OD 0x653000 thru 0x653fff (0x1000 bytes) 000376[0] 1779068625.379620 mapread 0x324000 thru 0x329c52 (0x5c53 bytes) 000378[1] 1779068625.415638 trunc from 0x85b794 to 0x5e5922 (0x275e71 bytes) 000379[1] 1779068625.503774 trunc from 0x5e5923 to 0x27ed26 (0x366bfc bytes) 000384[1] 1779068625.593710 trunc from 0x27ed27 to 0x891ab2 (0x612d8c bytes) 000385[0] 1779068625.644328 write 0x58c000 thru 0x5917fb (0x57fc bytes) 000387[0] 1779068625.711781 read 0x452000 thru 0x453e92 (0x1e93 bytes) 000389[1] 1779068625.741955 mapread 0x039000 thru 0x03dfed (0x4fee bytes) 000390[0] 1779068625.831378 mapwrite 0x170000 thru 0x178819 (0x881a bytes) 000391[0] 1779068626.014651 read 0x364000 thru 0x365ad0 (0x1ad1 bytes) 000395[0] 1779068626.119641 mapread 0x60e000 thru 0x619998 (0xb999 bytes) 000397[1] 1779068626.191184 read_OD 0x037000 thru 0x041fff (0xb000 bytes) 000398[1] 1779068626.234572 mapwrite 0x639000 thru 0x644474 (0xb475 bytes) 000400[0] 1779068626.363634 write 0x132000 thru 0x13dad9 (0xbada bytes) 000401[1] 1779068626.383067 mapread 0x668000 thru 0x66de97 (0x5e98 bytes) 000404[1] 1779068626.450651 mapread 0x005000 thru 0x00a053 (0x5054 bytes) 000406[0] 1779068626.492387 mapwrite 0x0f8000 thru 0x0fd907 (0x5908 bytes) 000407[1] 1779068626.647522 write_OD 0x2a6000 thru 0x2a7fff (0x2000 bytes) 000408[0] 1779068626.711362 read 0x839000 thru 0x8472de (0xe2df bytes) 000409[0] 1779068626.725406 mapwrite 0x90e000 thru 0x9165f0 (0x85f1 bytes) 000410[1] 1779068626.879821 trunc from 0x9165f1 to 0x119429 (0x7fd1c7 bytes) 000411[0] 1779068626.968285 read 0x010000 thru 0x01f2c1 (0xf2c2 bytes) 000413[0] 1779068627.079507 mapread 0x0f3000 thru 0x0f31a7 (0x1a8 bytes) 000414[1] 1779068627.081197 mapwrite 0x6b7000 thru 0x6c1657 (0xa658 bytes) 000418[0] 1779068627.328606 read 0x05f000 thru 0x06ea79 (0xfa7a bytes) 000419[0] 1779068627.362508 mapread 0x2c0000 thru 0x2ca945 (0xa946 bytes) 000420[1] 1779068627.394698 trunc from 0x6c1658 to 0x839704 (0x1780ad bytes) 000427[0] 1779068627.505495 mapread 0x3ab000 thru 0x3b069a (0x569b bytes) 000431[1] 1779068627.570253 trunc from 0x839705 to 0x8848a8 (0x4b1a4 bytes) 000432[1] 1779068627.659528 read_OD 0x842000 thru 0x844fff (0x3000 bytes) 000434[1] 1779068627.680656 write_OD 0x515000 thru 0x51efff (0xa000 bytes) 000435[1] 1779068627.741462 mapwrite 0x890000 thru 0x895c1b (0x5c1c bytes) 000437[1] 1779068627.907198 mapwrite 0x016000 thru 0x01ebe4 (0x8be5 bytes) 000440[0] 1779068628.114995 trunc from 0x895c1c to 0x112415 (0x783806 bytes) 000441[0] 1779068628.159611 write 0x331000 thru 0x332e25 (0x1e26 bytes) 000443[0] 1779068628.202451 mapwrite 0x4d5000 thru 0x4dedb6 (0x9db7 bytes) 000444[1] 1779068628.342434 write_OD 0x231000 thru 0x235fff (0x5000 bytes) 000446[1] 1779068628.414735 mapwrite 0x2c8000 thru 0x2d1cdb (0x9cdc bytes) 000458[1] 1779068628.667288 mapwrite 0x35f000 thru 0x36cd1c (0xdd1d bytes) 000460[0] 1779068628.761848 mapwrite 0x42a000 thru 0x42b508 (0x1509 bytes) 000461[1] 1779068628.966587 read_OD 0x1da000 thru 0x1e8fff (0xf000 bytes) 000465[0] 1779068629.042624 read 0x176000 thru 0x17d2d2 (0x72d3 bytes) 000467[1] 1779068629.062231 read_OD 0x4c5000 thru 0x4ccfff (0x8000 bytes) 000470[0] 1779068629.087615 mapwrite 0x1b4000 thru 0x1b9c37 (0x5c38 bytes) 000471[0] 1779068629.299559 read 0x1c8000 thru 0x1c8abb (0xabc bytes) 000474[1] 1779068629.338803 mapwrite 0x9ed000 thru 0x9f4a28 (0x7a29 bytes) 000475[0] 1779068629.557961 write 0x504000 thru 0x50910c (0x510d bytes) 000476[0] 1779068629.591568 write 0x011000 thru 0x017b0a (0x6b0b bytes) 000478[1] 1779068629.655053 mapwrite 0x7b4000 thru 0x7c06c5 (0xc6c6 bytes) 000482[1] 1779068629.851342 write_OD 0x56f000 thru 0x571fff (0x3000 bytes) 000484[1] 1779068629.963728 trunc from 0x9f4a29 to 0x3add84 (0x646ca4 bytes) 000485[1] 1779068630.042076 read_OD 0x13d000 thru 0x13dfff (0x1000 bytes) 000488[0] 1779068630.114168 read 0x020000 thru 0x024177 (0x4178 bytes) 000490[0] 1779068630.201402 trunc from 0x3add85 to 0x5db323 (0x22d59f bytes) 000492[0] 1779068630.385380 mapread 0x5c4000 thru 0x5d1434 (0xd435 bytes) 000493[1] 1779068630.480536 read_OD 0x1a1000 thru 0x1a2fff (0x2000 bytes) 000494[1] 1779068630.519156 read_OD 0x4d8000 thru 0x4e4fff (0xd000 bytes) 000495[1] 1779068630.540445 mapread 0x266000 thru 0x26eeae (0x8eaf bytes) 000496[0] 1779068630.568632 write 0x45e000 thru 0x45e21f (0x220 bytes) 000497[1] 1779068630.595071 read_OD 0x50a000 thru 0x50cfff (0x3000 bytes) 000498[1] 1779068630.625991 write_OD 0x1b4000 thru 0x1b5fff (0x2000 bytes) 000500[0] 1779068630.697694 mapread 0x01c000 thru 0x024590 (0x8591 bytes) All operations completed A-OK! PASS 16a (57s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:43:57 (1779068637) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.457733 s, 18.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.352227 s, 23.8 MB/s Chance of close/open is 1 in 50 Seed set to 8640 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000006[1] 1779068641.480242 mapwrite 0x7d3172 thru 0x7d4c1a (0x1aa9 bytes) 000007[0] 1779068641.620661 mapread 0x000648 thru 0x001de1 (0x179a bytes) 000010[0] 1779068641.734796 read 0x2f10af thru 0x2f1da8 (0xcfa bytes) 000012[1] 1779068641.783371 write 0x736240 thru 0x73700f (0xdd0 bytes) 000016[0] 1779068641.900260 write 0x4fabf0 thru 0x4fcb3e (0x1f4f bytes) truncating to largest ever: 0x750493 000017[0] 1779068641.954573 trunc from 0x7d4c1b to 0x750492 (0x84788 bytes) 000019[0] 1779068642.089793 write 0x20d8b9 thru 0x20e59c (0xce4 bytes) 000024[1] 1779068642.204700 read 0x5c98d6 thru 0x5ca19d (0x8c8 bytes) 000025[0] 1779068642.264856 mapwrite 0x0ddc21 thru 0x0df0bb (0x149b bytes) 000027[0] 1779068642.366149 write 0x7c88ac thru 0x7c9ee6 (0x163b bytes) 000028[0] 1779068642.382668 mapwrite 0x887edc thru 0x888c54 (0xd79 bytes) 000029[1] 1779068642.488323 read 0x34aa38 thru 0x34b4dd (0xaa6 bytes) 000030[1] 1779068642.524314 mapread 0x447280 thru 0x448940 (0x16c1 bytes) 000036[0] 1779068642.583850 write 0x19bff2 thru 0x19c9aa (0x9b9 bytes) 000038[0] 1779068642.635085 trunc from 0x888c55 to 0x3ceca3 (0x4b9fb1 bytes) 000039[0] 1779068642.737268 mapread 0x2336c5 thru 0x2337a3 (0xdf bytes) 000040[1] 1779068642.776228 read 0x1c010e thru 0x1c11c1 (0x10b4 bytes) 000045[0] 1779068643.086038 trunc from 0x3ceca4 to 0x3663c8 (0x688db bytes) 000046[1] 1779068643.184592 trunc from 0x3663c9 to 0x69e58a (0x3381c2 bytes) 000049[1] 1779068643.309977 read 0x320cf2 thru 0x320e61 (0x170 bytes) 000055[0] 1779068643.483376 mapwrite 0x713e0f thru 0x714e6e (0x1060 bytes) 000056[0] 1779068643.680550 read 0x102cb0 thru 0x102dd0 (0x121 bytes) 000064[1] 1779068643.831147 mapread 0x2204ea thru 0x222141 (0x1c58 bytes) 000067[1] 1779068643.959758 read 0x4ccaed thru 0x4cdd5c (0x1270 bytes) 000068[1] 1779068644.014431 write 0x9661bd thru 0x967eee (0x1d32 bytes) 000070[0] 1779068644.065194 trunc from 0x967eef to 0x48b36a (0x4dcb84 bytes) 000071[1] 1779068644.186604 mapwrite 0x4ae886 thru 0x4afbe5 (0x1360 bytes) 000073[0] 1779068644.430988 read 0x42bb93 thru 0x42bf61 (0x3cf bytes) 000074[0] 1779068644.468512 write 0x3831b3 thru 0x384b86 (0x19d4 bytes) 000080[0] 1779068644.579413 mapwrite 0x06d304 thru 0x06d7f4 (0x4f1 bytes) 000082[0] 1779068644.700110 mapwrite 0x44da0b thru 0x44e12d (0x723 bytes) 000085[1] 1779068644.788646 trunc from 0x4afbe6 to 0x74507d (0x295498 bytes) 000086[1] 1779068644.870599 read 0x02fd1f thru 0x030e95 (0x1177 bytes) truncating to largest ever: 0x8df3ec 000087[1] 1779068644.912070 trunc from 0x74507e to 0x8df3eb (0x19a36e bytes) 000090[0] 1779068645.025701 read 0x5cae93 thru 0x5cb5e6 (0x754 bytes) truncating to largest ever: 0x9e7835 000091[1] 1779068645.091448 trunc from 0x8df3ec to 0x9e7834 (0x108449 bytes) 000094[1] 1779068645.337663 read 0x76757d thru 0x76778f (0x213 bytes) 000095[0] 1779068645.383043 trunc from 0x9e7835 to 0x837683 (0x1b01b1 bytes) 000096[1] 1779068645.437679 write 0x99dc05 thru 0x99fb4a (0x1f46 bytes) 000098[1] 1779068645.511461 mapread 0x0b6a42 thru 0x0b721b (0x7da bytes) 000100[1] 1779068645.633762 write 0x7daa5f thru 0x7dc9fd (0x1f9f bytes) 000102[0] 1779068645.704512 write 0x2300d3 thru 0x230c76 (0xba4 bytes) 000103[1] 1779068645.790117 mapread 0x729aab thru 0x72ad18 (0x126e bytes) 000105[0] 1779068645.860359 read 0x96a275 thru 0x96adb7 (0xb43 bytes) 000109[1] 1779068645.931376 trunc from 0x99fb4b to 0x5f2ba2 (0x3acfa8 bytes) 000110[1] 1779068645.975440 trunc from 0x5f2ba3 to 0x4dbd50 (0x116e52 bytes) 000113[0] 1779068646.000132 read 0x11cb5b thru 0x11d6e2 (0xb88 bytes) 000115[1] 1779068646.043576 mapwrite 0x1be02c thru 0x1be396 (0x36b bytes) 000116[0] 1779068646.137585 mapwrite 0x882037 thru 0x8836c1 (0x168b bytes) 000117[1] 1779068646.322343 write 0x3ce87a thru 0x3ced3b (0x4c2 bytes) 000118[1] 1779068646.389845 write 0x33eb55 thru 0x33ee85 (0x331 bytes) 000121[0] 1779068646.485302 mapread 0x519335 thru 0x51a385 (0x1051 bytes) 000122[1] 1779068646.530614 mapwrite 0x9df075 thru 0x9dfa59 (0x9e5 bytes) 000126[0] 1779068646.706233 write 0x47c181 thru 0x47d5dc (0x145c bytes) 000128[0] 1779068646.767951 trunc from 0x9dfa5a to 0x63e768 (0x3a12f1 bytes) 000132[0] 1779068646.932205 read 0x4335bb thru 0x434b69 (0x15af bytes) 000133[1] 1779068646.976345 mapread 0x5bf06b thru 0x5c0c4f (0x1be5 bytes) 000134[1] 1779068647.044249 trunc from 0x63e769 to 0x18b956 (0x4b2e12 bytes) 000135[1] 1779068647.090573 read 0x011cf2 thru 0x01317e (0x148d bytes) 000137[0] 1779068647.170050 trunc from 0x18b957 to 0x0e7420 (0xa4536 bytes) 000138[1] 1779068647.324689 trunc from 0x0e7421 to 0x157f4d (0x70b2d bytes) 000139[1] 1779068647.439061 mapread 0x0b6c27 thru 0x0b7abb (0xe95 bytes) 000141[0] 1779068647.511821 mapread 0x12f994 thru 0x131788 (0x1df5 bytes) 000142[0] 1779068647.629330 mapread 0x140459 thru 0x1409d9 (0x581 bytes) 000143[1] 1779068647.637454 read 0x154e37 thru 0x156d50 (0x1f1a bytes) 000145[1] 1779068647.843473 mapwrite 0x28b7c4 thru 0x28b847 (0x84 bytes) 000146[0] 1779068648.100477 read 0x03a23d thru 0x03aa3d (0x801 bytes) 000147[0] 1779068648.260192 read 0x061b30 thru 0x0625db (0xaac bytes) 000148[0] 1779068648.301654 trunc from 0x28b848 to 0x7f8d6e (0x56d527 bytes) 000149[0] 1779068648.438123 trunc from 0x7f8d6f to 0x6368c5 (0x1c24a9 bytes) 000152[0] 1779068648.572929 write 0x0dcf00 thru 0x0dda25 (0xb26 bytes) 000154[1] 1779068648.709229 mapread 0x627bb2 thru 0x629061 (0x14b0 bytes) 000156[1] 1779068648.757334 write 0x571021 thru 0x571f96 (0xf76 bytes) 000157[1] 1779068648.791170 trunc from 0x6368c6 to 0x2063d0 (0x4304f5 bytes) 000158[0] 1779068648.838058 mapwrite 0x4faf1e thru 0x4fb0ca (0x1ad bytes) 000159[1] 1779068649.035244 mapwrite 0x0e69b8 thru 0x0e7a43 (0x108c bytes) 000164[1] 1779068649.413393 mapwrite 0x1c1b54 thru 0x1c1fcc (0x479 bytes) 000167[1] 1779068649.679425 mapwrite 0x5232cb thru 0x524af5 (0x182b bytes) 000168[1] 1779068649.951584 write 0x238834 thru 0x238c82 (0x44f bytes) 000170[0] 1779068649.997759 trunc from 0x524af6 to 0x724c2b (0x200136 bytes) 000171[0] 1779068650.064912 mapwrite 0x287646 thru 0x28954b (0x1f06 bytes) 000172[0] 1779068650.199952 mapread 0x1d232e thru 0x1d2d31 (0xa04 bytes) 000173[0] 1779068650.232295 mapwrite 0x89b98e thru 0x89d935 (0x1fa8 bytes) 000175[1] 1779068650.450398 write 0x676c91 thru 0x678a21 (0x1d91 bytes) 000176[0] 1779068650.611471 trunc from 0x89d936 to 0x12cc1c (0x770d19 bytes) 000178[0] 1779068650.732172 mapwrite 0x2c85b0 thru 0x2c8d15 (0x766 bytes) 000181[0] 1779068650.833900 trunc from 0x2c8d16 to 0x63477c (0x36ba67 bytes) 000182[0] 1779068650.881301 mapwrite 0x89d1ba thru 0x89ebf9 (0x1a40 bytes) 000184[1] 1779068651.008766 trunc from 0x89ebfa to 0x81e619 (0x805e0 bytes) 000185[1] 1779068651.094148 trunc from 0x81e61a to 0x05d5b3 (0x7c1066 bytes) 000186[0] 1779068651.123381 write 0x5ea094 thru 0x5ebc6e (0x1bdb bytes) 000190[1] 1779068651.219558 write 0x47e696 thru 0x47fa44 (0x13af bytes) 000199[1] 1779068651.400638 write 0x62392e thru 0x62499f (0x1072 bytes) 000200[0] 1779068651.421167 trunc from 0x6249a0 to 0x0fa162 (0x52a83d bytes) 000201[1] 1779068651.519121 write 0x928127 thru 0x929179 (0x1053 bytes) 000204[1] 1779068651.646316 mapwrite 0x5abc01 thru 0x5abce9 (0xe9 bytes) 000205[1] 1779068651.724453 mapread 0x071572 thru 0x073373 (0x1e02 bytes) 000209[1] 1779068651.842315 trunc from 0x92917a to 0x667d53 (0x2c1426 bytes) 000213[1] 1779068651.938200 trunc from 0x667d54 to 0x52fb1d (0x138236 bytes) 000215[1] 1779068652.045553 trunc from 0x52fb1e to 0x66caac (0x13cf8f bytes) 000219[1] 1779068652.171976 mapread 0x4a40ef thru 0x4a5749 (0x165b bytes) 000223[0] 1779068652.250098 write 0x4c7f93 thru 0x4c918f (0x11fd bytes) 000224[0] 1779068652.289253 trunc from 0x66caad to 0x4cb142 (0x1a196a bytes) 000226[0] 1779068652.437244 trunc from 0x4cb143 to 0x87d69c (0x3b255a bytes) 000227[1] 1779068652.532448 mapwrite 0x724d3e thru 0x726b1d (0x1de0 bytes) 000229[1] 1779068652.737177 trunc from 0x87d69d to 0x267134 (0x616568 bytes) 000231[1] 1779068652.858292 mapwrite 0x74eb59 thru 0x74ed47 (0x1ef bytes) 000232[0] 1779068652.994216 mapread 0x6577a6 thru 0x65791e (0x179 bytes) 000234[0] 1779068653.072142 trunc from 0x74ed48 to 0x15adf7 (0x5f3f50 bytes) 000237[0] 1779068653.165490 mapread 0x018d08 thru 0x01a3df (0x16d8 bytes) 000238[0] 1779068653.185965 write 0x474f34 thru 0x475c3b (0xd08 bytes) 000239[0] 1779068653.214558 write 0x5ee477 thru 0x5ef74f (0x12d9 bytes) 000240[0] 1779068653.249929 read 0x48cddd thru 0x48dc43 (0xe67 bytes) 000241[1] 1779068653.319447 trunc from 0x5ef750 to 0x53d052 (0xb26fd bytes) 000245[0] 1779068653.513117 mapread 0x2db78f thru 0x2dcaa3 (0x1315 bytes) 000246[0] 1779068653.572296 mapread 0x0c234d thru 0x0c35c7 (0x127b bytes) 000247[0] 1779068653.633271 write 0x04b1bd thru 0x04ce31 (0x1c75 bytes) 000248[1] 1779068653.686418 write 0x511660 thru 0x511875 (0x216 bytes) 000250[0] 1779068653.734463 trunc from 0x53d053 to 0x43a5cf (0x102a83 bytes) 000252[0] 1779068653.870605 read 0x069892 thru 0x06b399 (0x1b08 bytes) 000254[0] 1779068653.962256 read 0x2853fd thru 0x285987 (0x58b bytes) 000256[1] 1779068654.002401 mapread 0x3b1acd thru 0x3b1b21 (0x55 bytes) 000258[0] 1779068654.121031 write 0x486b0c thru 0x486fe9 (0x4de bytes) 000259[1] 1779068654.126497 mapread 0x25ddf8 thru 0x25de12 (0x1b bytes) 000261[1] 1779068654.191503 trunc from 0x486fea to 0x5e5cde (0x15ecf5 bytes) 000266[1] 1779068654.305350 trunc from 0x5e5cdf to 0x347e17 (0x29dec7 bytes) 000270[0] 1779068654.432421 mapwrite 0x3748db thru 0x375859 (0xf7f bytes) 000274[0] 1779068654.646335 read 0x01610c thru 0x016387 (0x27c bytes) 000277[1] 1779068654.736613 read 0x09c0b0 thru 0x09d422 (0x1373 bytes) 000281[1] 1779068654.982619 trunc from 0x37585a to 0x6ac4e1 (0x336c88 bytes) 000287[0] 1779068655.161378 trunc from 0x6ac4e2 to 0x7cab12 (0x11e631 bytes) 000288[0] 1779068655.271261 read 0x08b7c6 thru 0x08ca08 (0x1243 bytes) 000289[1] 1779068655.288603 trunc from 0x7cab13 to 0x73b592 (0x8f580 bytes) 000290[1] 1779068655.447242 write 0x34edba thru 0x34f22a (0x471 bytes) 000291[0] 1779068655.494623 mapwrite 0x46a64a thru 0x46c4ff (0x1eb6 bytes) 000294[0] 1779068655.774047 mapread 0x41cfcb thru 0x41e0f8 (0x112e bytes) 000296[0] 1779068655.880537 mapwrite 0x784735 thru 0x7861b9 (0x1a85 bytes) 000297[0] 1779068656.214277 mapwrite 0x190b8b thru 0x1922c7 (0x173d bytes) 000300[1] 1779068656.457640 trunc from 0x7861ba to 0x8a3356 (0x11d19d bytes) 000304[1] 1779068656.620240 mapwrite 0x0a9437 thru 0x0aa97a (0x1544 bytes) 000308[0] 1779068656.740630 mapread 0x2d8023 thru 0x2d8fc8 (0xfa6 bytes) 000309[1] 1779068656.862542 read 0x53f3aa thru 0x53f882 (0x4d9 bytes) 000310[1] 1779068656.899710 read 0x727f8e thru 0x7294d7 (0x154a bytes) 000311[0] 1779068656.976214 mapwrite 0x9d43c2 thru 0x9d59cf (0x160e bytes) 000315[1] 1779068657.254373 write 0x7add63 thru 0x7afcaa (0x1f48 bytes) 000316[1] 1779068657.305485 mapread 0x70f202 thru 0x710e6b (0x1c6a bytes) 000319[1] 1779068657.402594 trunc from 0x9d59d0 to 0x17804c (0x85d983 bytes) 000321[1] 1779068657.570623 mapwrite 0x01df60 thru 0x01ef5e (0xfff bytes) 000326[1] 1779068657.826087 mapwrite 0x6e758d thru 0x6e7fe4 (0xa58 bytes) 000327[0] 1779068657.955339 trunc from 0x6e7fe5 to 0x04dffb (0x699fe9 bytes) 000331[0] 1779068658.047551 trunc from 0x04dffc to 0x4392db (0x3eb2e0 bytes) 000332[1] 1779068658.118782 write 0x7816e9 thru 0x7836d8 (0x1ff0 bytes) 000333[0] 1779068658.156261 mapread 0x15a8bf thru 0x15bed0 (0x1612 bytes) 000334[0] 1779068658.223122 write 0x99cd29 thru 0x99e80a (0x1ae2 bytes) 000335[1] 1779068658.254834 write 0x2fc762 thru 0x2fcc70 (0x50f bytes) 000338[1] 1779068658.425827 trunc from 0x99e80b to 0x5d0488 (0x3ce382 bytes) 000339[0] 1779068658.512847 read 0x340ee5 thru 0x342b8c (0x1ca8 bytes) 000341[0] 1779068658.563577 trunc from 0x5d0489 to 0x696d0c (0xc6884 bytes) 000343[1] 1779068658.662628 read 0x09e0df thru 0x09f732 (0x1654 bytes) 000345[0] 1779068658.729019 mapread 0x5713c2 thru 0x571b05 (0x744 bytes) 000347[0] 1779068658.776350 mapwrite 0x7ef21c thru 0x7ef6a4 (0x489 bytes) 000352[1] 1779068658.982408 trunc from 0x7ef6a5 to 0x5abddb (0x2438c9 bytes) 000353[1] 1779068659.096947 mapread 0x1bbdc2 thru 0x1bd147 (0x1386 bytes) 000354[0] 1779068659.154466 mapread 0x11474c thru 0x115932 (0x11e7 bytes) 000356[1] 1779068659.242191 mapwrite 0x415e2f thru 0x417d5c (0x1f2e bytes) 000357[1] 1779068659.400686 write 0x98b897 thru 0x98d893 (0x1ffd bytes) 000358[0] 1779068659.488456 mapwrite 0x4b3aef thru 0x4b52b8 (0x17ca bytes) 000359[1] 1779068659.711914 mapread 0x339937 thru 0x33a8d6 (0xfa0 bytes) 000360[1] 1779068659.775159 write 0x88eb35 thru 0x88f3bc (0x888 bytes) 000361[1] 1779068659.808061 trunc from 0x98d894 to 0x96f4d1 (0x1e3c2 bytes) 000362[1] 1779068659.861117 mapwrite 0x229a7a thru 0x22a6c2 (0xc49 bytes) 000364[1] 1779068660.018233 trunc from 0x96f4d2 to 0x28cf87 (0x6e254a bytes) 000367[0] 1779068660.049149 read 0x1d9d78 thru 0x1db5e8 (0x1871 bytes) 000368[0] 1779068660.068089 mapread 0x072e53 thru 0x073bef (0xd9d bytes) 000370[0] 1779068660.177586 write 0x1a787a thru 0x1a970d (0x1e94 bytes) 000371[0] 1779068660.269187 trunc from 0x28cf88 to 0x935e32 (0x6a8eab bytes) 000372[1] 1779068660.405337 trunc from 0x935e33 to 0x143b0c (0x7f2326 bytes) 000373[0] 1779068660.577928 write 0x57a885 thru 0x57ac52 (0x3ce bytes) 000377[0] 1779068660.825539 write 0x2fa78d thru 0x2fb6eb (0xf5f bytes) 000382[1] 1779068660.914759 trunc from 0x57ac53 to 0x4c955d (0xb16f5 bytes) 000383[1] 1779068660.972374 trunc from 0x4c955e to 0x65d425 (0x193ec8 bytes) 000386[0] 1779068661.037978 trunc from 0x65d426 to 0x5a14c3 (0xbbf62 bytes) 000388[0] 1779068661.182226 mapread 0x597e9c thru 0x598f5f (0x10c4 bytes) 000389[0] 1779068661.216811 mapwrite 0x0da898 thru 0x0dad29 (0x492 bytes) 000390[1] 1779068661.306569 trunc from 0x5a14c4 to 0x8e7249 (0x345d86 bytes) 000391[0] 1779068661.393992 read 0x737fec thru 0x7395aa (0x15bf bytes) 000395[1] 1779068661.502537 mapwrite 0x4ba530 thru 0x4bc0fb (0x1bcc bytes) 000398[1] 1779068661.825341 mapwrite 0x80b4da thru 0x80d075 (0x1b9c bytes) 000400[1] 1779068662.085971 trunc from 0x8e724a to 0x13ef0d (0x7a833c bytes) 000405[0] 1779068662.241582 mapwrite 0x277358 thru 0x27809a (0xd43 bytes) 000407[0] 1779068662.483997 mapread 0x1d555e thru 0x1d6375 (0xe18 bytes) 000409[1] 1779068662.507634 write 0x690cca thru 0x692826 (0x1b5d bytes) 000410[1] 1779068662.588486 write 0x241630 thru 0x24315c (0x1b2d bytes) 000412[0] 1779068662.635537 mapread 0x099ebe thru 0x09b5db (0x171e bytes) 000414[0] 1779068662.724176 mapwrite 0x2eaf27 thru 0x2eb4b8 (0x592 bytes) 000415[1] 1779068662.854150 mapread 0x31e3cc thru 0x31eda1 (0x9d6 bytes) 000417[1] 1779068662.921100 write 0x6e5ade thru 0x6e66e4 (0xc07 bytes) 000418[0] 1779068662.948177 write 0x6a2fe5 thru 0x6a4f6b (0x1f87 bytes) 000420[1] 1779068662.978740 read 0x1bcc95 thru 0x1bd552 (0x8be bytes) 000421[1] 1779068662.997069 write 0x2ca1d0 thru 0x2ca7d4 (0x605 bytes) 000423[1] 1779068663.052596 mapwrite 0x74ebe4 thru 0x74f50a (0x927 bytes) 000425[1] 1779068663.211287 read 0x5175be thru 0x517a15 (0x458 bytes) 000426[1] 1779068663.226601 read 0x4f6b23 thru 0x4f8205 (0x16e3 bytes) 000428[0] 1779068663.280082 read 0x2f9fb3 thru 0x2fb2e7 (0x1335 bytes) 000429[0] 1779068663.324618 write 0x42e8de thru 0x42f329 (0xa4c bytes) 000430[1] 1779068663.371133 mapread 0x307688 thru 0x3085d7 (0xf50 bytes) 000434[1] 1779068663.427619 mapread 0x011f76 thru 0x012051 (0xdc bytes) 000436[0] 1779068663.449960 mapread 0x5aeb9e thru 0x5b09c3 (0x1e26 bytes) 000438[1] 1779068663.563761 mapwrite 0x45e009 thru 0x45e545 (0x53d bytes) 000443[0] 1779068663.828525 write 0x51a09a thru 0x51bdf7 (0x1d5e bytes) 000445[0] 1779068663.916542 mapwrite 0x8dfe43 thru 0x8e0bc5 (0xd83 bytes) 000447[1] 1779068664.143548 read 0x3cb54a thru 0x3cc43c (0xef3 bytes) 000449[1] 1779068664.191731 mapread 0x8dd3b6 thru 0x8deeb7 (0x1b02 bytes) 000451[0] 1779068664.233220 trunc from 0x8e0bc6 to 0x62ad50 (0x2b5e75 bytes) 000452[1] 1779068664.336445 mapwrite 0x13e106 thru 0x13ff9d (0x1e98 bytes) 000453[0] 1779068664.491065 write 0x5ed092 thru 0x5eeb45 (0x1ab4 bytes) 000456[0] 1779068664.588338 trunc from 0x62ad51 to 0x2b70dc (0x373c74 bytes) 000457[0] 1779068664.636480 trunc from 0x2b70dd to 0x86eb8b (0x5b7aaf bytes) 000459[0] 1779068664.668356 write 0x9e24b8 thru 0x9e3f69 (0x1ab2 bytes) 000464[1] 1779068664.796196 trunc from 0x9e3f6a to 0x8e553c (0xfea2d bytes) 000466[1] 1779068664.897418 trunc from 0x8e553d to 0x8b8343 (0x2d1f9 bytes) 000468[1] 1779068664.979352 trunc from 0x8b8344 to 0x7cec1f (0xe9724 bytes) 000469[1] 1779068665.052798 read 0x383f99 thru 0x385c5d (0x1cc5 bytes) 000471[1] 1779068665.125849 read 0x4116f3 thru 0x41180e (0x11c bytes) 000475[1] 1779068665.263772 read 0x32d291 thru 0x32e9b4 (0x1724 bytes) 000476[1] 1779068665.311953 read 0x623a31 thru 0x624620 (0xbf0 bytes) 000482[0] 1779068665.430534 write 0x88f60d thru 0x88fb8b (0x57f bytes) 000484[0] 1779068665.496636 mapwrite 0x4500cf thru 0x451a56 (0x1988 bytes) 000488[1] 1779068665.689534 mapwrite 0x487c8f thru 0x4889f4 (0xd66 bytes) 000489[1] 1779068665.909663 mapread 0x407784 thru 0x408368 (0xbe5 bytes) 000490[0] 1779068666.001789 trunc from 0x88fb8c to 0x4e0b5d (0x3af02e bytes) 000493[1] 1779068666.084976 write 0x100306 thru 0x100595 (0x290 bytes) 000497[0] 1779068666.289848 read 0x3c4f00 thru 0x3c624f (0x1350 bytes) 000499[0] 1779068666.368405 write 0x9f0704 thru 0x9f0b1b (0x418 bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:44:33 (1779068673) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:44:37 (1779068677) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.603618 s, 13.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.329905 s, 25.4 MB/s ldlm.namespaces.MGC192.168.203.153@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c41100ee000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c4110d45800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c41100ee000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c41100ee000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.7553 s, 6.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0972 s, 20.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.94073 s, 54.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.40658 s, 16.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.62679 s, 12.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.37453 s, 31.1 MB/s PASS 16d (54s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:45:31 (1779068731) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.268809 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698808 s, 15.0 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:45:39 (1779068739) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:46:06 (1779068766) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 21:46:34 (1779068794) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 21:46:40 (1779068800) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 21:46:46 (1779068806) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04955 s, 12.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20698 s, 21.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3186 s, 11.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.35353 s, 19.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.00787 s, 13.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74222 s, 15.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38827 s, 11.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.37458 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38591 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07172 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24487 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.90025 s, 13.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.16731 s, 12.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3721 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.49848 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.00469 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.29513 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.91939 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03489 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (34s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:47:20 (1779068840) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Sun May 17 21:47:57 EDT 2026 + fsx 2 done: rc=0 Sun May 17 21:47:57 EDT 2026 + fsx 3 done: rc=0 Sun May 17 21:47:57 EDT 2026 + fsx 4 done: rc=0 Sun May 17 21:47:57 EDT 2026 + fsx 5 done: rc=0 Sun May 17 21:47:57 EDT 2026 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:48:03 (1779068883) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 21:48:12 (1779068892) mmap test1: basic mmap operation (PASS, 0.091057s) mmap test2: MAP_PRIVATE not write back (PASS, 0.07574s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.597s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3901s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15938s) mmap test6: check mmap write/read content on two nodes (PASS, 0.2306s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1015s) mmap test11: multi-thread shared mmap (PASS, 0.90018s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0743s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:48:42 (1779068922) oleg353-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:48:46 (1779068926) PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:48:51 (1779068931) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (4s) == sanityn test 23: others should see updated atime while another read============================================================== 21:48:55 (1779068935) now is 1779068937 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 new atime is 1779068998 PASS 23 (66s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:50:01 (1779069001) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (4s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:50:05 (1779069005) PASS 24b (4s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:50:09 (1779069009) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (4s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:50:13 (1779069013) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 21:50:15 (1779069015) utime: good mknod times 1779069015 <= 1779069016 <= 1779069016 for /mnt/lustre/f26a utime: good mknod times 1779069015 <= 1779069016 <= 1779069016 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (5s) == sanityn test 26b: sync mtime between ost and mds ====== 21:50:20 (1779069020) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:50:27 (1779069027) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 21:50:28.000000000 -0400 Birth: 2026-05-17 21:50:27.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 21:50:28.000000000 -0400 Birth: 2026-05-17 21:50:27.000000000 -0400 PASS 26c (5s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:50:33 (1779069033) 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:50:39 (1779069039) fail_loc=0x314 PASS 31a (5s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:50:44 (1779069044) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.153@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (16s) == sanityn test 31r: open-rename(replace) race =========== 21:51:00 (1779069060) -rw-r--r-- 1 root root 0 May 17 21:51 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (7s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:51:08 (1779069068) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42924 Terminated tail -f $tfile PASS 31s (5s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:51:13 (1779069073) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:51:18 (1779069078) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:51:20 (1779069080) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:51:22 (1779069082) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:51:24 (1779069084) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 21:51:26 (1779069086) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid in FULL state after 0 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid in IDLE state after 0 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid in FULL state after 0 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid in IDLE state after 0 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c41100ee000.ost_server_uuid in FULL state after 0 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c41100ee000.ost_server_uuid in IDLE state after 0 sec PASS 34 (101s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:53:07 (1779069187) fail_loc=0x80000317 Race attempt 0 Wait for 49043 49136 for 60 sec... - open/close 1049 (time 1779069201.84 total 10.00 last 104.87) - open/close 2156 (time 1779069211.84 total 20.00 last 110.69) - open/close 3393 (time 1779069221.84 total 30.01 last 123.67) total: 4000 open/close in 35.61 seconds: 112.34 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.153@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.153@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (68s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:54:15 (1779069255) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.711 s, 73.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.58362 s, 89.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.55832 s, 93.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.605197 s, 86.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.796996 s, 65.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.663999 s, 79.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.507862 s, 103 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.48049 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.535151 s, 98.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.62761 s, 83.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.490149 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7492608) after_dd(7445504) after(7496704) PASS 36 (234s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 21:58:09 (1779069489) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 - create 2867 (time 1779069501.45 total 10.00 last 286.68) - create 7091 (time 1779069511.45 total 20.00 last 422.31) total: 10000 create in 28.91 seconds: 345.89 ops/second PASS 37 (34s) == sanityn test 39a: file mtime does not change after rename ========================================================== 21:58:43 (1779069523) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:58:46 (1779069526) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 21:58:50 (1779069530) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 21:58:54 (1779069534) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 21:58:57 (1779069537) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (11s) == sanityn test 40c: pdirops: link and others ======================================================================== 21:59:08 (1779069548) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 21:59:18 (1779069558) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (10s) == sanityn test 40e: pdirops: rename and others ======================================================================== 21:59:28 (1779069568) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (9s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 21:59:37 (1779069577) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 21:59:43 (1779069583) open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 21:59:50 (1779069590) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 21:59:56 (1779069596) Conflict PASS 41d (7s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:00:03 (1779069603) Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:00:09 (1779069609) Conflict PASS 41f (7s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:00:16 (1779069616) Conflict PASS 41g (7s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:00:23 (1779069623) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 22:00:29 (1779069629) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (745s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:12:54 (1779070374) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:13:00 (1779070380) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:13:06 (1779070386) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:13:12 (1779070392) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:13:17 (1779070397) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:13:23 (1779070403) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:13:29 (1779070409) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:13:35 (1779070415) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:13:40 (1779070420) PASS 43a (22s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:14:02 (1779070442) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:14:07 (1779070447) Conflict PASS 43c (6s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:14:13 (1779070453) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:14:19 (1779070459) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:14:24 (1779070464) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:14:29 (1779070469) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:14:35 (1779070475) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:14:40 (1779070480) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:14:41 (1779070481) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (45s) == sanityn test 43k: unlink vs create ==================== 22:15:26 (1779070526) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (460s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:23:06 (1779070986) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun May 17 22:23:10 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:23:11 (1779070991) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:23:16 (1779070996) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:23:22 (1779071002) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:23:27 (1779071007) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:23:32 (1779071012) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:23:38 (1779071018) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:23:43 (1779071023) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:23:48 (1779071028) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:23:49 (1779071029) PASS 45a (28s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:24:17 (1779071057) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:24:23 (1779071063) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:24:28 (1779071068) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:24:34 (1779071074) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:24:39 (1779071079) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:24:45 (1779071085) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:24:50 (1779071090) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:24:55 (1779071095) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 22:24:56 (1779071096) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (461s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 22:32:37 (1779071557) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 22:32:42 (1779071562) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 22:32:48 (1779071568) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 22:32:53 (1779071573) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 22:32:58 (1779071578) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 22:33:03 (1779071583) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 22:33:09 (1779071589) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 22:33:14 (1779071594) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 22:33:19 (1779071599) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 22:33:20 (1779071600) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 22:33:21 (1779071601) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 22:33:22 (1779071602) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 22:33:23 (1779071603) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 22:33:24 (1779071604) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 22:33:25 (1779071605) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 22:33:26 (1779071606) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 22:33:27 (1779071607) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00485883 s, 2.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 22:33:34 (1779071614) 0+1 records in 0+1 records out 158 bytes copied, 0.00365076 s, 43.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (259827) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 22:33:38 (1779071618) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00509814 s, 201 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144838 s, 72.4 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 22:33:52 (1779071632) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00387818 s, 264 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 22:33:59 (1779071639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156556 s, 67.0 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 22:34:02 (1779071642) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 22:34:06 (1779071646) llite.lustre-ffff9c41100ee000.xattr_cache=1 llite.lustre-ffff9c4110d45800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:07.000000000 -0400 Modify: 2026-05-17 22:34:07.000000000 -0400 Change: 2026-05-17 22:34:07.000000000 -0400 Birth: 2026-05-17 22:34:06.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:12.000000000 -0400 Modify: 2026-05-17 22:34:13.000000000 -0400 Change: 2026-05-17 22:34:13.000000000 -0400 Birth: 2026-05-17 22:34:12.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:18.000000000 -0400 Modify: 2026-05-17 22:34:18.000000000 -0400 Change: 2026-05-17 22:34:18.000000000 -0400 Birth: 2026-05-17 22:34:17.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:18.000000000 -0400 Modify: 2026-05-17 22:34:18.000000000 -0400 Change: 2026-05-17 22:34:18.000000000 -0400 Birth: 2026-05-17 22:34:17.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:23.000000000 -0400 Modify: 2026-05-17 22:34:24.000000000 -0400 Change: 2026-05-17 22:34:24.000000000 -0400 Birth: 2026-05-17 22:34:23.000000000 -0400 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:34:23.000000000 -0400 Modify: 2026-05-17 22:34:24.000000000 -0400 Change: 2026-05-17 22:34:24.000000000 -0400 Birth: 2026-05-17 22:34:23.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 22:34:30 (1779071670) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 22:34:38 (1779071678) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 22:34:45 (1779071685) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 22:34:58 (1779071698) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 22:35:07 (1779071707) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 22:35:08 (1779071708) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 22:35:21 (1779071721) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 22:35:35 (1779071735) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1779071736 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.90662 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (24s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 22:35:59 (1779071759) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1779071760 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.69922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (37s) == sanityn test 60: Verify data_version behaviour ======== 22:36:36 (1779071796) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 22:36:38 (1779071798) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 22:36:41 (1779071801) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 22:36:44 (1779071804) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0031849 s, 20.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 22:36:45 (1779071805) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00296013 s, 22.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 22:36:46 (1779071806) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 22:36:47 (1779071807) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00392188 s, 16.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 22:36:48 (1779071808) llite.lustre-ffff9c41100ee000.xattr_cache=1 llite.lustre-ffff9c4110d45800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 22:36:50 (1779071810) llite.lustre-ffff9c41100ee000.xattr_cache=1 llite.lustre-ffff9c4110d45800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 22:36:52 (1779071812) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00502084 s, 204 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00304293 s, 337 kB/s 272584: get lock2 272586: get lock1 272584: done 272584: get lock1 272586: done 272586: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 272586: cannot get lock2: Resource deadlock avoided 272586: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 272584: cannot get lock1: Resource deadlock avoided 272584: done 272584: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 22:36:58 (1779071818) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0219447 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139778 s, 75.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273167 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 22:37:05 (1779071825) opening files: ........ closing files: ........ PASS 76 (18s) == sanityn test 77a: check FIFO NRS policy =============== 22:37:23 (1779071843) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130433 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032701 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520207 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717549 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596399 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130098 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114792 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128659 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141498 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156307 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155291 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165886 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148065 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145491 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15036 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139132 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170492 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804724 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587992 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296725 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251395 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01767 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285483 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254217 s, 41.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157274 s, 66.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130778 s, 80.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899165 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162695 s, 64.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020838 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340805 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017405 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239999 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00194078 s, 540 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 22:37:25 (1779071845) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134024 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293386 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288693 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114517 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12604 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112383 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118246 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12349 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100576 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135662 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129625 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119226 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123462 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139518 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143381 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132118 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135474 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310021 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234606 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270704 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312957 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284396 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220183 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328967 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870542 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365402 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033289 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295379 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158808 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213283 s, 49.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320181 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152009 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154259 s, 68.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119898 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301755 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385241 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466463 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326249 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753256 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813301 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993574 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108955 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122775 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121181 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125265 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109838 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11697 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126455 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1241 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145416 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213004 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160684 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264288 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232832 s, 45.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107325 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171564 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220652 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131322 s, 79.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235973 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371923 s, 282 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236554 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465864 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212312 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194878 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010932 s, 95.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141353 s, 74.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 22:37:29 (1779071849) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149849 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336083 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372753 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489238 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466917 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631812 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726705 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646738 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768672 s, 13.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092309 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10136 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884033 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117747 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104083 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101712 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115643 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128114 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185371 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215925 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189746 s, 55.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253221 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204938 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286996 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125307 s, 83.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125224 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293113 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169777 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022305 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268063 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300344 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536213 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00555 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00270464 s, 388 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126924 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388942 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639488 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497665 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624213 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612147 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083386 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868156 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750389 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674725 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723929 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753067 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107613 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091602 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103664 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110854 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891292 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167977 s, 62.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015412 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155615 s, 67.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183545 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189306 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00684083 s, 153 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286959 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472146 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241359 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254694 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290559 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0088598 s, 118 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00636292 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271772 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130564 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499506 s, 210 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12547 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280425 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167871 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784015 s, 13.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832956 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712774 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869796 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102339 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963035 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102938 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910448 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972886 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11879 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119352 s, 8.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958613 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866369 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12427 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019538 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208755 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305875 s, 343 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133633 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324589 s, 323 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222071 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172366 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990173 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247369 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207412 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00317664 s, 330 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195298 s, 53.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160512 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925352 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127755 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226259 s, 46.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 22:37:34 (1779071854) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121486 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295772 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448351 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121753 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019464 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143511 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152229 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213433 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016125 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243693 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973903 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816327 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340909 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690707 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226988 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035062 s, 299 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297316 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00518449 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199691 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147516 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163187 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157787 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160893 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164833 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206032 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242059 s, 4.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242085 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219353 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232725 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238303 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239772 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231247 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282242 s, 3.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153098 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452858 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069005 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723656 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798955 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722785 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707506 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952211 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650385 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864139 s, 12.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125274 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120269 s, 8.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137614 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160125 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146551 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163582 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15218 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012845 s, 81.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014387 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217988 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230602 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260978 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188182 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246655 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196267 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012126 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00906074 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587382 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136381 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129529 s, 81.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021965 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820726 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458947 s, 228 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130716 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264274 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252037 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637598 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777095 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847797 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930681 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937094 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869979 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085479 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763734 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848564 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894643 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100316 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894284 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734412 s, 14.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102217 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637678 s, 164 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015616 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128722 s, 81.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694794 s, 151 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124932 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186195 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176824 s, 59.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373815 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252951 s, 415 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214734 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425306 s, 247 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574407 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490644 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00316327 s, 331 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407502 s, 257 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302842 s, 346 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 22:37:39 (1779071859) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.53@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161317 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021834 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272932 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492211 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628398 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423904 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641206 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741325 s, 14.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931598 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08633 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767329 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810004 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796326 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989825 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10277 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104927 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886859 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108509 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104034 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11496 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104735 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107172 s, 9.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110978 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106278 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110821 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114465 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111108 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111676 s, 9.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110989 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109171 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116281 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111687 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115775 s, 9.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135556 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213716 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035232 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387738 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595037 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318449 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690069 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959052 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877478 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900532 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679387 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976188 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886502 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850477 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102644 s, 10.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107262 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951786 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885549 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980591 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921481 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101787 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094121 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930827 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950488 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972306 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985566 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101227 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997333 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992224 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108911 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103007 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103417 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103567 s, 10.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120974 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212507 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479066 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620895 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568658 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547902 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894259 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115886 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10348 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10611 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115622 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10825 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105896 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102898 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110001 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126288 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137432 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220935 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100117 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131481 s, 79.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265519 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169935 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184501 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208028 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133867 s, 78.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191833 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145798 s, 71.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302895 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670515 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328914 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182811 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723654 s, 145 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164381 s, 63.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116872 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164416 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275679 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064765 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809801 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646259 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964543 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967663 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865326 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111301 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997456 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118563 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119324 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100471 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108884 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11937 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102432 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123871 s, 84.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158696 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163469 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143719 s, 73.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413938 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155384 s, 67.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756506 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227154 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147706 s, 71.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124437 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203409 s, 51.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146853 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138011 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569607 s, 184 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00750745 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151834 s, 69.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 22:37:46 (1779071866) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23292 s, 72.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178014 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299926 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631664 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787479 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816256 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116912 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116967 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122305 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127514 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117095 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11086 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1282 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120399 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117504 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124042 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119446 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219027 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143118 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176623 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202001 s, 51.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254726 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169512 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248391 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363067 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243081 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385793 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295776 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268341 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608003 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308403 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367572 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110487 s, 94.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137331 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267586 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203167 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794111 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763326 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910456 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857193 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930458 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097496 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877195 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106554 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11312 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978868 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119371 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121417 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107598 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122421 s, 8.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380914 s, 275 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166951 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123764 s, 84.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160418 s, 65.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191991 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231372 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765641 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163021 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546492 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246329 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234413 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200588 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048314 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400479 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00792742 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302372 s, 34.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190013 s, 88.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691411 s, 15.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726031 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693647 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867166 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920915 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750011 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870943 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939667 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107258 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810727 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101742 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703665 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109112 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858125 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847345 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108184 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194067 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171199 s, 61.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364065 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299239 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218473 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034636 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334753 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335413 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323632 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136532 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140412 s, 74.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00893403 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168981 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161071 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305543 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151717 s, 69.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126103 s, 133 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176268 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764512 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769584 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897892 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887084 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872587 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775435 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760331 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085188 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957783 s, 10.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859523 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942602 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095803 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727556 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932022 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939205 s, 11.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228757 s, 45.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225763 s, 46.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142425 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198592 s, 52.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290026 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159239 s, 65.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220942 s, 47.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134563 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127661 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254821 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182094 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253979 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311795 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297109 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137986 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103571 s, 101 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 22:37:54 (1779071874) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 22:37:57 (1779071877) oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg353-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg353-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg353-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg353-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 22:38:01 (1779071881) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 22:38:08 (1779071888) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113405 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035453 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630599 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615674 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798117 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073683 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101701 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941065 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951448 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105435 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087935 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106798 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120164 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114374 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139689 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129621 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148213 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140636 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220271 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236627 s, 44.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209948 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226307 s, 46.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303223 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024049 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018893 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006369 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157233 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998792 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150413 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161741 s, 64.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278722 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330635 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225332 s, 46.5 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.07349 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.431 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (47s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 22:38:55 (1779071935) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22729 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118794 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126931 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13106 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114212 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136723 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120852 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115948 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107806 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151819 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151958 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168703 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15818 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139767 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176855 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175989 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159374 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232273 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237557 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221612 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223054 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261878 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210952 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00752414 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240113 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215229 s, 48.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253692 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288447 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895421 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268113 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401891 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532795 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043558 s, 241 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.837 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9514 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22647 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633148 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649542 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813626 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721363 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822624 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108399 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967882 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092912 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171725 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171741 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164689 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183397 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185395 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195224 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191326 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168728 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014685 s, 71.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315461 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333818 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109274 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372212 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160089 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226958 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177836 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166762 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436797 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205038 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214056 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253757 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428638 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239707 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496217 s, 21.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.834 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9492 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (135s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 22:41:10 (1779072070) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23099 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725482 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755301 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704264 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694864 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914423 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095038 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848436 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776952 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1582 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158402 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167065 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152526 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157888 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145468 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186865 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179196 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213907 s, 49.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213852 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224671 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232031 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021576 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234572 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163492 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023135 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243282 s, 43.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209242 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157352 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254633 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404274 s, 259 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254556 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457936 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500003 s, 210 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8361 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9471 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22547 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807188 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924337 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791502 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770234 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070527 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716218 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911018 s, 11.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102565 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159196 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159226 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145607 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158106 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147295 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171056 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174351 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186911 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142307 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182808 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124306 s, 84.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192864 s, 54.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156703 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176647 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246688 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256387 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215184 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01632 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413084 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747341 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139097 s, 75.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235156 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852205 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113652 s, 92.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8328 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9572 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 22:43:23 (1779072203) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255575 s, 65.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307695 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277307 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952416 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806578 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837726 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888248 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899123 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110367 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892952 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096185 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106137 s, 9.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111489 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096335 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117667 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126096 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101489 s, 10.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171172 s, 61.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795927 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012732 s, 82.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137453 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253708 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243206 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253552 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195255 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197508 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00763763 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826407 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290211 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270125 s, 38.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238494 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285228 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630231 s, 166 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94325 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73533 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134747 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253912 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702347 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662085 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460625 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478799 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786334 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710795 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650167 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749735 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933255 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887453 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636396 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11075 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818555 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113481 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106792 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196681 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236507 s, 44.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225638 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677471 s, 155 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267389 s, 39.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122144 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207951 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262528 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00873576 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311799 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015012 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139695 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411467 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377926 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122948 s, 85.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019016 s, 55.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92971 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73642 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.11918 s, 141 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338376 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392003 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933843 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12278 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129075 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119862 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118752 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109587 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145376 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137691 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120793 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143149 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137741 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137158 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12243 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145579 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160042 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179886 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205583 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016568 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177529 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234142 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236886 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218151 s, 48.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249048 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183845 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115192 s, 91.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125074 s, 83.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0060652 s, 173 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104846 s, 100 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229233 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00955117 s, 110 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04767 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.888 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139005 s, 121 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254955 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468864 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693051 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531991 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768929 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075889 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910038 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994488 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102459 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100195 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089755 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925718 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959385 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528158 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964656 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730995 s, 14.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014247 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190385 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169807 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159582 s, 65.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160502 s, 65.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018204 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977795 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681948 s, 154 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174762 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236579 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251299 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238808 s, 43.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475338 s, 221 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799391 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399686 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00461611 s, 227 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78204 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7383 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91393 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73233 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.843403 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039022 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540099 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737737 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776742 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909428 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801116 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805332 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923113 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144038 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153917 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166221 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216731 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197497 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204842 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213618 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2275 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162985 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219179 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258305 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209686 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219733 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235834 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287985 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227326 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119464 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985449 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282508 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824811 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307301 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116843 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194926 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106873 s, 98.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6705 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4354 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271575 s, 61.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413383 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294505 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878338 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113145 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130195 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162307 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137396 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143644 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14959 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172347 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164517 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140332 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157394 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171934 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150702 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181108 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226559 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193719 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212565 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211845 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00995335 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114167 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109786 s, 95.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347001 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030719 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020633 s, 50.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151366 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421694 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133659 s, 78.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410261 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317528 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135068 s, 77.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93885 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73336 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122741 s, 137 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409872 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342214 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12218 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110791 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143611 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150733 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14402 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158555 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150768 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15537 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164806 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155664 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147123 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189604 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154258 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172765 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191629 s, 54.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190937 s, 54.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155075 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245329 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193351 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225347 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201164 s, 52.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223462 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210119 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268419 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231887 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280775 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067797 s, 155 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235022 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776774 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690758 s, 152 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77796 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73524 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77145 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7351 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (360s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 22:49:23 (1779072563) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256739 s, 65.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240422 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040868 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874532 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915298 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920902 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925323 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105557 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107031 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917946 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754346 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860593 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095363 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122194 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109036 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146796 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162934 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112901 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162493 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146863 s, 71.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253468 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019552 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142301 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228436 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234538 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199917 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203832 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578792 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281677 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679511 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962088 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593407 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00197906 s, 530 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03016 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73852 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 22:50:00 (1779072600) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151572 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294265 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296119 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106196 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106187 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808773 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969344 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105564 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930211 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929246 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098118 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871816 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823102 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109316 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100262 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945253 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117425 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201946 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208038 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168291 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202504 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291262 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274999 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162961 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309289 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154697 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131286 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187533 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304902 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272198 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362929 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656535 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251776 s, 41.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93947 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73242 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (37s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 22:50:37 (1779072637) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260049 s, 64.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402093 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390755 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739523 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888502 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104768 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101317 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109445 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986382 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103627 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976596 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945027 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946671 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949828 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104079 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122857 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110965 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163616 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140694 s, 74.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015721 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168756 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190102 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150817 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813603 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024857 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251498 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156637 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108333 s, 96.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116303 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00619033 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00461286 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268695 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289056 s, 363 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98018 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97579 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (31s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 22:51:08 (1779072668) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139711 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216881 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381824 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370573 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563953 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614606 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678283 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806553 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091143 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968583 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887424 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958893 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870846 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879021 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096701 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870459 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966779 s, 10.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142328 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016676 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139937 s, 74.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177148 s, 59.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202404 s, 51.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248346 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249935 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212937 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160635 s, 65.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927679 s, 113 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811013 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409258 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236162 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110741 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255472 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00205161 s, 511 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76884 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73506 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95312 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73599 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 22:52:25 (1779072745) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.844889 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254457 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430391 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060728 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541187 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532965 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740744 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694743 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656531 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162237 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197177 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185576 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194499 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189312 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202878 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21334 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227279 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015075 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189391 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234591 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194693 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240104 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232499 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239392 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025482 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168848 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275348 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704706 s, 149 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444906 s, 236 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266211 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406515 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135557 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360693 s, 291 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.668 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9591 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (64s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 22:53:29 (1779072809) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248337 s, 67.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322465 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074689 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853629 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762377 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941459 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867594 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956806 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135309 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155224 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18284 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166941 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168841 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177542 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169949 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19357 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189929 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023266 s, 45.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219006 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203833 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252138 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276496 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299483 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190431 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429598 s, 244 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120777 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170796 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293255 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121152 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156056 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334539 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157204 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310861 s, 337 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.967 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73094 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg353-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129796 s, 129 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341849 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363588 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216635 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102848 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931417 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116669 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10597 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103939 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114055 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108393 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10434 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101603 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114303 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993399 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103038 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102867 s, 10.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140106 s, 74.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126812 s, 82.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206264 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197555 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255311 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248485 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160753 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218807 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028757 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221442 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705931 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021287 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033662 s, 312 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235717 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235888 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248407 s, 42.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg353-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg353-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84311 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73874 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76594 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73263 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (115s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 22:55:25 (1779072925) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 22:55:28 (1779072928) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg353-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.49723 s, 300 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg353-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0258 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 22:56:18 (1779072978) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136535 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247938 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591953 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585774 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540914 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734888 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783987 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905292 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802305 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945531 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c4110d45800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108279 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906841 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110382 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107073 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11205 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108644 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116636 s, 9.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c4110d45800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209321 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152757 s, 68.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173873 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227373 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149184 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172903 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242038 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160089 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421415 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242421 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221527 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023145 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233592 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257306 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029049 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248023 s, 42.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97483 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96693 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.109887 s, 153 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515693 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504058 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657131 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466736 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686355 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800174 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112401 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115842 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138397 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137826 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14562 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136684 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136767 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132099 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129208 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135595 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012602 s, 83.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679997 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160577 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351684 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146619 s, 71.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145375 s, 72.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228138 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404801 s, 25.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203362 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375123 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297731 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364241 s, 28.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439279 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018683 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247698 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025339 s, 414 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg353-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg353-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97235 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96828 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 22:57:23 (1779073043) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 22:57:27 (1779073047) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (38s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 22:58:05 (1779073085) mds.MDS.mdt.nrs_policies=tbf oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg353-client: oleg353-server: ssh exited with exit code 36 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg353-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg353-client: oleg353-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 22:58:17 (1779073097) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 309165 kill md_thread xargs PASS 77r (41s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 22:58:58 (1779073138) oleg353-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg353-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 22:59:01 (1779073141) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506542 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:59:02.000000000 -0400 Modify: 2026-05-17 22:59:02.000000000 -0400 Change: 2026-05-17 22:59:02.000000000 -0400 Birth: 2026-05-17 22:59:01.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 22:59:14 (1779073154) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 22:59:15 (1779073155) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (0s) == sanityn test 81a: rename and stat under striped directory ========================================================== 22:59:16 (1779073156) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (0s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 22:59:17 (1779073157) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (0s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 22:59:18 (1779073158) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 22:59:18 (1779073158) total: 2500 open/close in 5.47 seconds: 456.93 ops/second snapshot_time 1779073181.421868573 secs.nsecs start_time 1779073159.292076530 secs.nsecs elapsed_time 22.129792043 secs.nsecs open 2505 samples [usecs] 76 4908 753928 313547776 close 2504 samples [usecs] 63 7917 273460 118239964 mknod 2501 samples [usecs] 172 4642 657198 253597178 unlink 2500 samples [usecs] 278 16590 1259906 2580676726 mkdir 2 samples [usecs] 790 1181 1971 2018861 rmdir 2 samples [usecs] 1637 3842 5479 17440733 rename 2500 samples [usecs] 252 19376 3182485 6699058395 getattr 3760 samples [usecs] 7 3307 77040 35595038 parallel_rename_file 2500 samples [usecs] 252 19376 3182485 6699058395 crossdir_rename 2500 samples [usecs] 252 19376 3182485 6699058395 PASS 81d (25s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 22:59:43 (1779073183) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7569 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7569 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 22:59:46 (1779073186) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 22:59:47 (1779073187) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 22:59:54 (1779073194) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1779073194 Finishing test test1 at 1779073195 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 22:59:57 (1779073197) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 22:59:58 (1779073198) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 22:59:59 (1779073199) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:00:00 (1779073200) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5901 0x170d 0x240000400 1 5874 0x16f2 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5902 0x170e 0x240000400 1 5875 0x16f3 0x280000400 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 23:00:08 (1779073208) osc.lustre-OST0000-osc-ffff9c41100ee000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9c4110d45800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c41100ee000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c4110d45800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0437932 s, 187 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09481 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 327814 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0126944 s, 323 kB/s osc.lustre-OST0000-osc-ffff9c41100ee000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c4110d45800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c41100ee000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c4110d45800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:00:18 (1779073218) llite.lustre-ffff9c41100ee000.fast_read=0 llite.lustre-ffff9c4110d45800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0151542 s, 138 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9c41100ee000.fast_read=1 llite.lustre-ffff9c4110d45800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:00:33 (1779073233) llite.lustre-ffff9c41100ee000.fast_read=0 llite.lustre-ffff9c4110d45800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00168078 s, 7.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9c41100ee000.fast_read=1 llite.lustre-ffff9c4110d45800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:00:41 (1779073241) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:00:42 (1779073242) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00152431 s, 2.7 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:00:44 (1779073244) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0478612 s, 43.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:00:46 (1779073246) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0519431 s, 40.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 23:00:48 (1779073248) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00157712 s, 7.6 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 23:00:50 (1779073250) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000393883 s, 10.4 MB/s PASS 101a (1s) == sanityn test 101b: Discard DoM data on rename ========= 23:00:51 (1779073251) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000433785 s, 9.4 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 23:00:53 (1779073253) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00040807 s, 10.0 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 23:00:57 (1779073257) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1779073257 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1779073260 Finishing test test23 at 1779073260 PASS 103 (6s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:01:03 (1779073263) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 23:01:04 (1779073264) fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 23:01:21 (1779073281) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:01:22 (1779073282) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00268777 s, 381 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272509077 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:01:23.000000000 -0400 Modify: 2026-05-17 23:01:23.000000000 -0400 Change: 2026-05-17 23:01:23.000000000 -0400 Birth: 2026-05-17 23:01:22.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272509077 -rw-r--r-- 1 0 0 1779073282 1779073283 1779073283 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272509077 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:01:23.000000000 -0400 Modify: 2026-05-17 23:01:23.000000000 -0400 Change: 2026-05-17 23:01:23.000000000 -0400 Birth: 2026-05-17 23:01:22.000000000 -0400 1779073283 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 23:01:24 (1779073284) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 23:01:26 (1779073286) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0723333 s, 145 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5881 0x16f9 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:01:30 (1779073290) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.075867 s, 138 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5911 0x1717 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7569 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 23:01:38 (1779073298) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0036461 s, 2.7 MB/s llite.lustre-ffff9c41100ee000.hybrid_io=0 llite.lustre-ffff9c4110d45800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.26231 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:01:44 (1779073304) 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre 192.168.203.153@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 PASS 109 (497s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:10:01 (1779073801) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:10:01.000000000 -0400 Modify: 2026-05-17 23:10:01.000000000 -0400 Change: 2026-05-17 23:10:01.000000000 -0400 Birth: 2026-05-17 23:10:01.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg353-client.virtnet oleg353-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:11:06 (1779073866) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 23:11:07 (1779073867) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 23:11:08 (1779073868) lustre-OST0000-osc-ffff9c41029c2000 active. lustre-OST0001-osc-ffff9c41029c2000 active. lustre-OST0000-osc-ffff9c41029c2000 active. lustre-OST0001-osc-ffff9c41029c2000 active. MGC192.168.203.153@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 23:11:11 (1779073871) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:11:13 (1779073873) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:11:15 (1779073875) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 23:11:16 (1779073876) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (5s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 23:11:21 (1779073881) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (2s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 23:11:23 (1779073883) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (1s) == sanityn test 117d: Per-User Trash can Type testing ==== 23:11:24 (1779073884) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 23:11:25 (1779073885) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (1s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 23:11:26 (1779073886) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 23:11:27 (1779073887) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 23:11:28 (1779073888) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/.ltrash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (4s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 23:11:32 (1779073892) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (2s) == sanityn test 117j: Move back file from Trash Can ====== 23:11:34 (1779073894) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 23:11:35 (1779073895) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 23:11:36 (1779073896) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (1s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 23:11:37 (1779073897) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (1s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 23:11:38 (1779073898) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (1s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 23:11:39 (1779073899) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (1s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 23:11:40 (1779073900) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 23:11:41 (1779073901) SKIP: sanityn test_117pb needs >= 2 MDTs SKIP 117pb (1s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 23:11:42 (1779073902) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (1s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 23:11:43 (1779073903) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (0s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 23:11:44 (1779073904) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (1s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 23:11:44 (1779073904) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (1s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 23:11:45 (1779073905) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (2s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 23:11:47 (1779073907) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 23:11:48 (1779073908) SKIP: sanityn test_117v ldiskfs only test SKIP 117v (1s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 23:11:49 (1779073909) SKIP: sanityn test_117w ldiskfs only test SKIP 117w (1s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 23:11:50 (1779073910) SKIP: sanityn test_117x ldiskfs only test SKIP 117x (1s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 23:11:51 (1779073911) SKIP: sanityn test_117ya ldiskfs only test SKIP 117ya (1s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 23:11:52 (1779073912) SKIP: sanityn test_117yb ldiskfs only test SKIP 117yb (1s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 23:11:53 (1779073913) SKIP: sanityn test_117yc ldiskfs only test SKIP 117yc (1s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 23:11:54 (1779073914) SKIP: sanityn test_117za ldiskfs only test SKIP 117za (1s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 23:11:55 (1779073915) SKIP: sanityn test_117zb ldiskfs only test SKIP 117zb (1s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 23:11:56 (1779073916) SKIP: sanityn test_117zc ldiskfs only test SKIP 117zc (1s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 23:11:57 (1779073917) On MGS 192.168.203.153, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.203.153, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] SKIP: sanityn test_117zd ldiskfs only test On MGS 192.168.203.153, active = nodemap.active=0 waiting 10 secs for sync SKIP 117zd (37s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 23:12:34 (1779073954) SKIP: sanityn test_118Aa ldiskfs only test SKIP 118Aa (2s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 23:12:36 (1779073956) SKIP: sanityn test_118Ab ldiskfs only test SKIP 118Ab (2s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 23:12:38 (1779073958) SKIP: sanityn test_118Ba ldiskfs only test SKIP 118Ba (2s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 23:12:40 (1779073960) SKIP: sanityn test_118Bb ldiskfs only test SKIP 118Bb (2s) == sanityn test 118C: TCU: add support for rename() ====== 23:12:42 (1779073962) SKIP: sanityn test_118C ldiskfs only test SKIP 118C (2s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 23:12:44 (1779073964) SKIP: sanityn test_118d ldiskfs only test SKIP 118d (2s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 23:12:47 (1779073967) SKIP: sanityn test_118e ldiskfs only test SKIP 118e (2s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 23:12:49 (1779073969) SKIP: sanityn test_118f ldiskfs only test SKIP 118f (2s) == sanityn test 118g: TCU: Tolerate ENOSPC error bypass Trash Can ========================================================== 23:12:51 (1779073971) SKIP: sanityn test_118g ldiskfs only test SKIP 118g (1s) == sanityn test 118h: TCU: run mdtest with TCU enabled === 23:12:52 (1779073972) SKIP: sanityn test_118h mdtest not found SKIP 118h (2s) == sanityn test 118i: TCU: trusted.unrm xattr should not be cached on client-side ========================================================== 23:12:54 (1779073974) SKIP: sanityn test_118i ldiskfs only test SKIP 118i (1s) == sanityn test 118j: TCU: access .Trash from users with different UID ========================================================== 23:12:55 (1779073975) SKIP: sanityn test_118j ldiskfs only test SKIP 118j (2s) == sanityn test 118k: TCU: acess .Trash from users with different NODEMAP UIDs ========================================================== 23:12:57 (1779073977) SKIP: sanityn test_118k ldiskfs only test SKIP 118k (2s) == sanityn test 118l: Access global trash dir via LU_TRASH_FID with subdir mount ========================================================== 23:12:59 (1779073979) SKIP: sanityn test_118l ldiskfs only test SKIP 118l (2s) == sanityn test 118m: Access .ltrash under normal root dir and subdir mount point ========================================================== 23:13:01 (1779073981) SKIP: sanityn test_118m ldiskfs only test SKIP 118m (1s) == sanityn test 118n: lfs trash state should return state for a symlink file correctly ========================================================== 23:13:02 (1779073982) SKIP: sanityn test_118n ldiskfs only test SKIP 118n (2s) == sanityn test 118o: lfs trash unrm should work with relative path ========================================================== 23:13:04 (1779073984) SKIP: sanityn test_118o ldiskfs only test SKIP 118o (1s) == sanityn test 118p: Setting notrash flag on symlink file ========================================================== 23:13:05 (1779073985) SKIP: sanityn test_118p ldiskfs only test SKIP 118p (2s) == sanityn test 118q: The mode of a directory moved into Trash should not be changed ========================================================== 23:13:07 (1779073987) SKIP: sanityn test_118q ldiskfs only test SKIP 118q (1s) == sanityn test 118r: TCU: non-root user access Trash Can and restore files ========================================================== 23:13:08 (1779073988) SKIP: sanityn test_118r ldiskfs only test SKIP 118r (2s) == sanityn test 118s: TCU: The FS flags of a file or directory should be kept ========================================================== 23:13:10 (1779073990) SKIP: sanityn test_118s ldiskfs only test SKIP 118s (2s) == sanityn test 118t: TCU: unrmed directory should keep LUSTRE_PROJHERIT_FL flag ========================================================== 23:13:12 (1779073992) SKIP: sanityn test_118t ldiskfs only test SKIP 118t (1s) == sanityn test 118u: TCU: Space usage of deleted files == 23:13:13 (1779073993) SKIP: sanityn test_118u ldiskfs only test SKIP 118u (2s) == sanityn test 118v: TCU: Trash Can for encrypted files ========================================================== 23:13:15 (1779073995) SKIP: sanityn test_118v ldiskfs only test SKIP 118v (1s) == sanityn test 118w: TCU: repeated deletion with same name for encrypted files ========================================================== 23:13:16 (1779073996) SKIP: sanityn test_118w ldiskfs only test SKIP 118w (2s) == sanityn test 118x: TCU: virtual .Trash for encrypted striped dir ========================================================== 23:13:18 (1779073998) SKIP: sanityn test_118x ldiskfs only test SKIP 118x (1s) == sanityn test 119a: ltrash_purge: verify dry-run mode and --empty mode ========================================================== 23:13:19 (1779073999) SKIP: sanityn test_119a ldiskfs only test SKIP 119a (2s) == sanityn test 119b: ltrash_purge: space-based purging with --auto-ost ========================================================== 23:13:21 (1779074001) SKIP: sanityn test_119b ldiskfs only test SKIP 119b (1s) == sanityn test 119c: ltrash_purge: age-based purging ==== 23:13:22 (1779074002) SKIP: sanityn test_119c ldiskfs only test SKIP 119c (2s) == sanityn test 119d: ltrash_purge: id-based filtering (UID/GID/PROJID) ========================================================== 23:13:24 (1779074004) SKIP: sanityn test_119d ldiskfs only test SKIP 119d (1s) == sanityn test 119e: ltrash_purge: verify --conf option ========================================================== 23:13:25 (1779074005) SKIP: sanityn test_119e ldiskfs only test SKIP 119e (2s) == sanityn test 119f: Run ltrash_purge on subdir mount === 23:13:27 (1779074007) SKIP: sanityn test_119f ldiskfs only test SKIP 119f (1s) == sanityn test 119g: ltrash_purge: parallel directory deletion ========================================================== 23:13:28 (1779074008) SKIP: sanityn test_119g ldiskfs only test SKIP 119g (2s) == sanityn test 200: service remains healthy while able to process request ========================================================== 23:13:30 (1779074010) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=20 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0879 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0925 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0964 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1126 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0902 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0994 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0945 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0908 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.095 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1442 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1303 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1073 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1138 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1135 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1629 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1185 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1468 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1495 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1912 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.241 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1876 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2338 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1843 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1864 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1971 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2624 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2079 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2969 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3082 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3368 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3256 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3235 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.354 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3038 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2874 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3096 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.336 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.39 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.362 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.41 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c41029c2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c41029c2000.ost_server_uuid in FULL state after 0 sec PASS 200 (107s) cleanup: ====================================================== == sanityn test complete, duration 5988 sec ============== 23:15:18 (1779074118) === sanityn: start cleanup 23:15:18 (1779074118) === Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg353-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 23:16:06 (1779074166) ===