-----============= acceptance-small: sanityn ============----- Fri Apr 3 07:38:35 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg431-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 07:38:58 (1775216338) === Starting client oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Mount client oleg431-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Started clients oleg431-client.virtnet: 192.168.204.131@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg431-client.virtnet: executing check_config_client /mnt/lustre oleg431-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg431-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a3182cf2000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a318850d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a3182cf2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a318850d000.idle_timeout=debug disable quota as required oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 07:39:36 (1775216376) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872858 s, 12.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7563] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 07:39:39 (1775216379) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 07:39:48 (1775216388) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (10s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 07:39:58 (1775216398) -rw-r--r-- 1 root root 0 Apr 3 07:40 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (10s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 07:40:08 (1775216408) -rw-r--r-- 1 root root 0 Apr 3 07:40 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (10s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 07:40:18 (1775216418) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 07:40:27 (1775216427) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 07:40:35 (1775216435) total 1 -rwxrwxrwx 1 root root 0 Apr 3 07:40 f2a -rwxrwxrwx 1 root root 0 Apr 3 07:40 f2b total 1 -rwxrwxrwx 1 root root 0 Apr 3 07:40 f2a -rwxrwxrwx 1 root root 0 Apr 3 07:40 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0178326 s, 28.7 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 07:40:43 (1775216443) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 07:40:47 (1775216447) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.449108 s, 4.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 07:40:56 (1775216456) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 07:41:03 (1775216463) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 07:41:11 (1775216471) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 07:41:20 (1775216480) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 07:41:28 (1775216488) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 07:41:34 (1775216494) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 07:41:42 (1775216502) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 07:41:51 (1775216511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555216 s, 18.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.114094 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 07:41:59 (1775216519) 1+0 records in 1+0 records out 1 byte copied, 0.0143307 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0182878 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.037852 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0247868 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108869 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0150605 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145423 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118109 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0190468 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011341 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0225165 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170823 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 07:42:08 (1775216528) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0164119 s, 187 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0738153 s, 55.5 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00240254 s, 1.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 07:42:15 (1775216535) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 07:42:22 (1775216542) warning: '-runtime' deprecated, use '-t runtime' instead - link 297 (time 1775216555.08 total 10.02 last 29.64) - link 581 (time 1775216565.10 total 20.05 last 28.33) - link 833 (time 1775216575.11 total 30.05 last 25.19) - link 1098 (time 1775216585.15 total 40.09 last 26.39) - link 1393 (time 1775216595.18 total 50.12 last 29.42) - link 1675 (time 1775216605.21 total 60.15 last 28.11) - link 1968 (time 1775216615.22 total 70.16 last 29.26) - link 2262 (time 1775216625.22 total 80.16 last 29.39) - link 2540 (time 1775216635.22 total 90.17 last 27.80) total: 2835 link in 100.02 seconds: 28.34 ops/second using seed 2879078685 running for 100 seconds - stat 4593 (time 1775216557 ; total 11 ; last 11) - stat 8770 (time 1775216568 ; total 22 ; last 11) - stat 10000 (time 1775216571 ; total 25 ; last 3) - stat 13888 (time 1775216582 ; total 36 ; last 11) - stat 18414 (time 1775216593 ; total 47 ; last 11) - stat 20000 (time 1775216597 ; total 51 ; last 4) - stat 21856 (time 1775216608 ; total 62 ; last 11) - stat 23895 (time 1775216619 ; total 73 ; last 11) - stat 25928 (time 1775216630 ; total 84 ; last 11) - stat 27885 (time 1775216641 ; total 95 ; last 11) total: 28864 stats in 100 seconds: 288.640015 stats/second - unlinked 0 (time 1775216596 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2835) error: No such file or directory total: 2834 unlinks in 90 seconds: 31.488890 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21369) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21371) - No such process PASS 12 (151s) == sanityn test 13: test directory page revocation ======= 07:44:53 (1775216693) f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 07:45:04 (1775216704) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22558 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (9s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 07:45:13 (1775216713) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23139 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 07:45:21 (1775216721) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23717 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 07:45:29 (1775216729) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24300 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 07:45:37 (1775216737) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24883 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 07:45:47 (1775216747) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 07:46:42 (1775216802) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.465025 s, 18.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.395929 s, 21.2 MB/s Chance of close/open is 1 in 50 Seed set to 6806 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[0] 1775216807.634362 mapwrite 0x74ed22 thru 0x7539bf (0x4c9e bytes) 000005[0] 1775216807.792326 write 0x0942b7 thru 0x09b42f (0x7179 bytes) 000007[0] 1775216807.812375 read 0x44332c thru 0x448ae8 (0x57bd bytes) 000008[0] 1775216807.826856 mapread 0x2dbbf8 thru 0x2dfdc2 (0x41cb bytes) 000009[0] 1775216807.861737 mapread 0x3341e3 thru 0x342370 (0xe18e bytes) 000012[1] 1775216807.975915 write 0x0288bc thru 0x029d42 (0x1487 bytes) 000014[0] 1775216808.073908 mapwrite 0x94e48e thru 0x95c956 (0xe4c9 bytes) 000016[1] 1775216808.397521 mapwrite 0x700eb3 thru 0x703655 (0x27a3 bytes) 000019[0] 1775216808.633193 write 0x5337bc thru 0x534bfc (0x1441 bytes) 000022[1] 1775216808.688989 write 0x3b9d63 thru 0x3c254d (0x87eb bytes) truncating to largest ever: 0x1a4a89 000023[1] 1775216808.726254 trunc from 0x95c957 to 0x1a4a88 (0x7b7ece bytes) 000027[0] 1775216808.845003 write 0x5cd7d3 thru 0x5d0b3a (0x3368 bytes) 000029[0] 1775216808.889049 read 0x331a28 thru 0x33ae61 (0x943a bytes) 000031[0] 1775216808.949145 read 0x22abaa thru 0x22ee1f (0x4276 bytes) 000032[1] 1775216808.999576 mapread 0x44c74e thru 0x44fd75 (0x3628 bytes) 000033[1] 1775216809.033381 mapread 0x3845f1 thru 0x3924c1 (0xded1 bytes) truncating to largest ever: 0x38bbbc 000034[1] 1775216809.136284 trunc from 0x5d0b3b to 0x38bbbb (0x244f7f bytes) 000035[0] 1775216809.231519 write 0x7f2e2c thru 0x7fcaa1 (0x9c76 bytes) 000036[0] 1775216809.269166 write 0x07fc79 thru 0x08061a (0x9a2 bytes) 000039[0] 1775216809.308215 mapwrite 0x0880dd thru 0x092ea7 (0xadcb bytes) truncating to largest ever: 0x423f59 000042[1] 1775216809.456253 trunc from 0x7fcaa2 to 0x423f58 (0x3d8b49 bytes) 000043[1] 1775216809.526410 write 0x7a9215 thru 0x7aa7f1 (0x15dd bytes) 000044[0] 1775216809.530176 read 0x61e1ef thru 0x62d5e2 (0xf3f4 bytes) 000047[1] 1775216809.610125 read 0x5b31e7 thru 0x5b6e72 (0x3c8c bytes) 000048[0] 1775216809.638629 write 0x305fd6 thru 0x310d65 (0xad90 bytes) 000050[1] 1775216809.693986 mapwrite 0x56fd03 thru 0x573fb2 (0x42b0 bytes) 000051[1] 1775216809.876075 read 0x745b54 thru 0x74d261 (0x770e bytes) 000052[0] 1775216809.911783 mapread 0x2a1811 thru 0x2a63ca (0x4bba bytes) 000054[1] 1775216809.990075 mapread 0x130c33 thru 0x13fc6f (0xf03d bytes) 000056[0] 1775216810.095267 read 0x72d095 thru 0x73c081 (0xefed bytes) 000057[1] 1775216810.132379 read 0x62102b thru 0x62d915 (0xc8eb bytes) 000058[1] 1775216810.142532 mapread 0x34523b thru 0x34fde7 (0xabad bytes) 000060[0] 1775216810.163120 read 0x6d9040 thru 0x6e545d (0xc41e bytes) 000063[1] 1775216810.175030 mapwrite 0x604454 thru 0x610780 (0xc32d bytes) 000064[1] 1775216810.355699 mapread 0x301835 thru 0x309b85 (0x8351 bytes) 000065[1] 1775216810.404351 mapread 0x777108 thru 0x7846e8 (0xd5e1 bytes) 000066[1] 1775216810.491782 mapread 0x2ed884 thru 0x2f3dba (0x6537 bytes) 000067[0] 1775216810.529001 write 0x51c136 thru 0x52730f (0xb1da bytes) 000069[0] 1775216810.599364 write 0x15ed45 thru 0x16a6c0 (0xb97c bytes) 000071[1] 1775216810.683144 write 0x44e9ce thru 0x44fb46 (0x1179 bytes) 000075[0] 1775216810.780685 mapwrite 0x35103c thru 0x353938 (0x28fd bytes) 000076[0] 1775216810.908578 read 0x31381a thru 0x31473f (0xf26 bytes) 000077[0] 1775216810.947538 mapread 0x510955 thru 0x5161c9 (0x5875 bytes) 000081[0] 1775216811.087666 mapwrite 0x3e63a0 thru 0x3f13a7 (0xb008 bytes) truncating to largest ever: 0x482e49 000082[0] 1775216811.219259 trunc from 0x7aa7f2 to 0x482e48 (0x3279a9 bytes) 000086[0] 1775216811.390948 read 0x23040e thru 0x236e0b (0x69fe bytes) 000087[1] 1775216811.410790 mapwrite 0x9ea1aa thru 0x9edc64 (0x3abb bytes) 000091[1] 1775216811.601678 read 0x2c35c7 thru 0x2c398b (0x3c5 bytes) 000092[0] 1775216811.608978 read 0x30eb93 thru 0x3104c1 (0x192f bytes) 000093[0] 1775216811.636373 mapread 0x5242e8 thru 0x530d39 (0xca52 bytes) 000095[0] 1775216811.701233 mapwrite 0x944e5e thru 0x952f48 (0xe0eb bytes) 000102[0] 1775216811.931323 mapread 0x797655 thru 0x79a274 (0x2c20 bytes) 000103[1] 1775216811.974868 mapwrite 0x46e6b0 thru 0x47a43a (0xbd8b bytes) 000104[0] 1775216812.245915 mapwrite 0x934aaf thru 0x93b2eb (0x683d bytes) 000107[0] 1775216812.400349 write 0x6ef0bc thru 0x6f5ae7 (0x6a2c bytes) 000109[1] 1775216812.426220 read 0x63a4dd thru 0x644b15 (0xa639 bytes) 000115[0] 1775216812.488937 trunc from 0x9edc65 to 0x4255d9 (0x5c868b bytes) 000117[0] 1775216812.598062 mapread 0x3a2ef7 thru 0x3aad7b (0x7e85 bytes) 000118[1] 1775216812.663063 mapread 0x07807f thru 0x07cdbf (0x4d41 bytes) 000120[1] 1775216812.798700 mapread 0x112b21 thru 0x11dc00 (0xb0e0 bytes) 000121[1] 1775216812.845606 mapread 0x3c27ae thru 0x3cdd0e (0xb561 bytes) 000125[0] 1775216812.948262 trunc from 0x4255da to 0x3c67b5 (0x5ee24 bytes) 000128[1] 1775216813.000959 write 0x679e1c thru 0x6801d8 (0x63bd bytes) 000132[1] 1775216813.097376 write 0x343409 thru 0x34d702 (0xa2fa bytes) 000133[1] 1775216813.185222 mapwrite 0x511757 thru 0x513448 (0x1cf2 bytes) 000134[0] 1775216813.297847 read 0x18b523 thru 0x1908e0 (0x53be bytes) truncating to largest ever: 0x921757 000135[0] 1775216813.381552 trunc from 0x6801d9 to 0x921756 (0x2a157e bytes) 000136[1] 1775216813.478003 write 0x6eb125 thru 0x6f7173 (0xc04f bytes) 000137[0] 1775216813.518588 mapread 0x6e29ec thru 0x6e6909 (0x3f1e bytes) 000138[0] 1775216813.619813 mapwrite 0x1dd543 thru 0x1e3bed (0x66ab bytes) 000139[0] 1775216813.784288 read 0x4c603c thru 0x4d03b6 (0xa37b bytes) 000140[1] 1775216813.858932 write 0x92d181 thru 0x93837d (0xb1fd bytes) 000144[0] 1775216813.974598 mapwrite 0x480426 thru 0x48749b (0x7076 bytes) 000150[0] 1775216814.427276 mapwrite 0x624688 thru 0x633344 (0xecbd bytes) 000151[1] 1775216814.688383 trunc from 0x93837e to 0x01a7d0 (0x91dbad bytes) 000152[0] 1775216814.824977 mapwrite 0x9dfd7b thru 0x9e4da2 (0x5028 bytes) 000153[0] 1775216815.201676 write 0x6f1532 thru 0x6fc5ef (0xb0be bytes) 000154[1] 1775216815.261225 mapwrite 0x77b246 thru 0x782090 (0x6e4b bytes) 000156[1] 1775216815.466014 mapread 0x0c9f4f thru 0x0cf3a9 (0x545b bytes) 000158[0] 1775216815.611032 read 0x0376d9 thru 0x0409d1 (0x92f9 bytes) 000159[1] 1775216815.625113 trunc from 0x9e4da3 to 0x892dab (0x151ff7 bytes) 000161[0] 1775216815.817848 mapread 0x0c5662 thru 0x0cc6f5 (0x7094 bytes) 000163[0] 1775216815.945313 mapread 0x44b406 thru 0x44e69d (0x3298 bytes) 000164[1] 1775216816.079255 read 0x61aaac thru 0x61e248 (0x379d bytes) 000167[0] 1775216816.103377 trunc from 0x892dac to 0x09455f (0x7fe84c bytes) 000168[1] 1775216816.209152 mapwrite 0x1f9607 thru 0x206187 (0xcb81 bytes) 000169[0] 1775216816.376241 mapwrite 0x3baf18 thru 0x3bc810 (0x18f9 bytes) 000172[1] 1775216816.561339 mapwrite 0x451fd0 thru 0x45e4c6 (0xc4f7 bytes) 000174[0] 1775216816.780789 trunc from 0x45e4c7 to 0x6ef6f1 (0x29122b bytes) 000175[1] 1775216816.910263 write 0x365f93 thru 0x36a382 (0x43f0 bytes) 000176[0] 1775216816.970029 mapwrite 0x0784d3 thru 0x085a9a (0xd5c8 bytes) 000177[1] 1775216817.187030 write 0x6cdd45 thru 0x6d6ba8 (0x8e64 bytes) 000178[0] 1775216817.263413 write 0x4918b8 thru 0x49a110 (0x8859 bytes) 000180[1] 1775216817.436877 mapwrite 0x8438cf thru 0x849dd1 (0x6503 bytes) 000184[0] 1775216817.802146 write 0x60bfc3 thru 0x611f24 (0x5f62 bytes) 000187[0] 1775216817.917962 read 0x1c12db thru 0x1c6511 (0x5237 bytes) 000188[1] 1775216817.940905 trunc from 0x849dd2 to 0x5f4781 (0x255650 bytes) 000189[1] 1775216818.017494 write 0x266b13 thru 0x2686a3 (0x1b91 bytes) 000190[0] 1775216818.035302 write 0x11c6cb thru 0x11d785 (0x10bb bytes) 000191[1] 1775216818.085308 trunc from 0x5f4782 to 0x440fae (0x1b37d3 bytes) 000196[0] 1775216818.244918 read 0x00979e thru 0x0169c1 (0xd224 bytes) 000197[1] 1775216818.277495 mapwrite 0x70e2b8 thru 0x7194c0 (0xb209 bytes) 000199[1] 1775216818.457328 trunc from 0x7194c1 to 0x7c6164 (0xacca4 bytes) 000200[1] 1775216818.523856 read 0x08e7b6 thru 0x09c22a (0xda75 bytes) 000204[0] 1775216818.616126 trunc from 0x7c6165 to 0x78ccab (0x394b9 bytes) 000205[0] 1775216818.726865 trunc from 0x78ccac to 0x1d96c7 (0x5b35e4 bytes) 000206[0] 1775216818.799697 mapwrite 0x4ae105 thru 0x4b5e22 (0x7d1e bytes) 000208[0] 1775216818.962275 read 0x3fdbc3 thru 0x403b2a (0x5f68 bytes) 000210[1] 1775216818.981666 write 0x218864 thru 0x21e82e (0x5fcb bytes) 000213[1] 1775216819.118872 mapwrite 0x836250 thru 0x83f7e9 (0x959a bytes) 000217[1] 1775216819.349162 write 0x4017ee thru 0x40bf49 (0xa75c bytes) 000218[1] 1775216819.376007 mapread 0x4b8ec0 thru 0x4c1ae5 (0x8c26 bytes) truncating to largest ever: 0x965a1e 000219[1] 1775216819.435380 trunc from 0x83f7ea to 0x965a1d (0x126234 bytes) 000220[0] 1775216819.491739 read 0x453046 thru 0x45efe5 (0xbfa0 bytes) 000224[0] 1775216819.641172 mapwrite 0x7a3dc9 thru 0x7a78c2 (0x3afa bytes) 000225[0] 1775216819.871296 write 0x92dcc8 thru 0x9392bd (0xb5f6 bytes) 000227[1] 1775216820.016553 read 0x38a5ab thru 0x39971f (0xf175 bytes) 000229[0] 1775216820.097212 mapwrite 0x916600 thru 0x925cb1 (0xf6b2 bytes) 000232[0] 1775216820.343687 trunc from 0x965a1e to 0x4ae503 (0x4b751a bytes) 000234[0] 1775216820.381410 trunc from 0x4ae504 to 0x774418 (0x2c5f15 bytes) 000235[1] 1775216820.406379 write 0x6c2321 thru 0x6d094a (0xe62a bytes) 000236[1] 1775216820.478827 mapread 0x53514e thru 0x538853 (0x3706 bytes) 000237[1] 1775216820.547771 read 0x3fb67c thru 0x405b6f (0xa4f4 bytes) 000238[1] 1775216820.593066 write 0x287493 thru 0x287a12 (0x580 bytes) 000239[0] 1775216820.621409 read 0x3a3e9a thru 0x3a6ee5 (0x304c bytes) 000240[0] 1775216820.715985 mapread 0x700ee3 thru 0x70398f (0x2aad bytes) 000245[1] 1775216820.867600 read 0x079b0e thru 0x07b3bc (0x18af bytes) 000246[0] 1775216820.876842 mapwrite 0x19e013 thru 0x1ad7da (0xf7c8 bytes) 000247[0] 1775216821.083025 write 0x7085d9 thru 0x70f9a8 (0x73d0 bytes) 000248[1] 1775216821.161378 write 0x0618b5 thru 0x06b7f1 (0x9f3d bytes) 000249[1] 1775216821.229282 mapwrite 0x6b77dd thru 0x6babc2 (0x33e6 bytes) 000258[1] 1775216821.610258 read 0x0df3fa thru 0x0e7102 (0x7d09 bytes) 000262[0] 1775216821.697205 read 0x252c0d thru 0x259be4 (0x6fd8 bytes) 000268[0] 1775216821.875528 read 0x3ee24b thru 0x3fd35f (0xf115 bytes) 000269[0] 1775216821.923413 mapread 0x0008c6 thru 0x005672 (0x4dad bytes) 000270[1] 1775216822.049196 mapread 0x31ef32 thru 0x3231b0 (0x427f bytes) 000272[0] 1775216822.105132 read 0x679bd8 thru 0x680e8c (0x72b5 bytes) 000273[1] 1775216822.151310 write 0x726aa6 thru 0x72f708 (0x8c63 bytes) 000274[0] 1775216822.207337 write 0x47ee72 thru 0x48dabf (0xec4e bytes) 000277[0] 1775216822.278802 mapwrite 0x72a705 thru 0x73a127 (0xfa23 bytes) 000280[1] 1775216822.494199 trunc from 0x774419 to 0x553606 (0x220e12 bytes) 000282[1] 1775216822.556334 mapread 0x4470e9 thru 0x44f76e (0x8686 bytes) 000283[1] 1775216822.616614 write 0x4d4a7f thru 0x4e3c7c (0xf1fe bytes) 000284[1] 1775216822.653966 write 0x119619 thru 0x11c35e (0x2d46 bytes) 000285[0] 1775216822.670005 mapread 0x493c80 thru 0x49c8f8 (0x8c79 bytes) 000287[1] 1775216822.809347 mapread 0x050510 thru 0x055c9d (0x578e bytes) 000288[1] 1775216822.843308 mapread 0x4e054b thru 0x4eea62 (0xe518 bytes) 000289[1] 1775216822.939706 read 0x33a242 thru 0x3430e7 (0x8ea6 bytes) 000290[0] 1775216822.995759 mapwrite 0x0ca679 thru 0x0cedb7 (0x473f bytes) 000292[1] 1775216823.149670 write 0x891070 thru 0x894589 (0x351a bytes) 000293[0] 1775216823.167181 mapwrite 0x56f59b thru 0x5782f7 (0x8d5d bytes) 000296[1] 1775216823.632268 write 0x1a181d thru 0x1a717c (0x5960 bytes) 000297[0] 1775216823.662974 read 0x3b02b0 thru 0x3bef4d (0xec9e bytes) 000299[0] 1775216823.750684 mapread 0x15adae thru 0x15e4c4 (0x3717 bytes) 000300[0] 1775216823.865912 mapread 0x5d81d8 thru 0x5e551a (0xd343 bytes) 000301[1] 1775216824.041732 read 0x448608 thru 0x44c22a (0x3c23 bytes) 000302[0] 1775216824.076531 read 0x17095d thru 0x1761f0 (0x5894 bytes) 000304[1] 1775216824.098056 trunc from 0x89458a to 0x727456 (0x16d133 bytes) 000305[0] 1775216824.210974 mapread 0x6925bf thru 0x69f77a (0xd1bc bytes) 000306[0] 1775216824.297271 read 0x4c921e thru 0x4d3157 (0x9f3a bytes) 000307[1] 1775216824.350978 trunc from 0x727457 to 0x1be9b3 (0x568aa3 bytes) 000309[1] 1775216824.423318 trunc from 0x1be9b4 to 0x8cd589 (0x70ebd6 bytes) 000312[1] 1775216824.524285 read 0x45d822 thru 0x46624f (0x8a2e bytes) 000314[1] 1775216824.597220 write 0x1c7195 thru 0x1d4210 (0xd07c bytes) 000316[0] 1775216824.631115 mapread 0x24c25b thru 0x25637d (0xa123 bytes) 000317[0] 1775216824.704062 mapread 0x04ec49 thru 0x05d1d3 (0xe58b bytes) 000321[0] 1775216824.807270 mapread 0x28a92a thru 0x28b38f (0xa66 bytes) 000322[0] 1775216824.857401 trunc from 0x8cd58a to 0x2d6361 (0x5f7228 bytes) 000326[0] 1775216824.944333 read 0x1954b0 thru 0x195f7e (0xacf bytes) 000328[1] 1775216825.114270 read 0x23bf51 thru 0x24a029 (0xe0d9 bytes) 000329[1] 1775216825.271240 mapread 0x159b6f thru 0x15f766 (0x5bf8 bytes) 000331[0] 1775216825.339811 trunc from 0x2d6362 to 0x0ccc92 (0x2096cf bytes) 000332[0] 1775216825.422173 mapwrite 0x69fd4b thru 0x6a6705 (0x69bb bytes) 000334[1] 1775216825.658114 write 0x790bf6 thru 0x7a0bbf (0xffca bytes) 000335[1] 1775216825.699966 mapwrite 0x1bb056 thru 0x1be832 (0x37dd bytes) 000336[0] 1775216825.880583 trunc from 0x7a0bc0 to 0x958071 (0x1b74b2 bytes) 000337[0] 1775216825.996353 write 0x49c49d thru 0x49c8cf (0x433 bytes) truncating to largest ever: 0x990f11 000341[1] 1775216826.085151 trunc from 0x958072 to 0x990f10 (0x38e9f bytes) 000342[1] 1775216826.175718 mapread 0x050e13 thru 0x05ac5e (0x9e4c bytes) 000344[0] 1775216826.401081 mapread 0x3b69e9 thru 0x3c2bff (0xc217 bytes) 000345[1] 1775216826.614024 mapwrite 0x7bc03a thru 0x7c3e39 (0x7e00 bytes) 000348[1] 1775216826.868103 read 0x7e4d5a thru 0x7f47f5 (0xfa9c bytes) 000349[1] 1775216826.888385 write 0x848570 thru 0x84e819 (0x62aa bytes) 000350[1] 1775216826.915988 mapwrite 0x8a4b37 thru 0x8b18ab (0xcd75 bytes) 000351[1] 1775216827.084172 mapread 0x450241 thru 0x450de2 (0xba2 bytes) 000354[0] 1775216827.116066 trunc from 0x990f11 to 0x102054 (0x88eebc bytes) 000355[1] 1775216827.214154 mapread 0x08a2f1 thru 0x09803d (0xdd4d bytes) 000356[0] 1775216827.335647 read 0x0e48c6 thru 0x0ee6cf (0x9e0a bytes) 000357[0] 1775216827.389823 mapwrite 0x9d2e8b thru 0x9df3d7 (0xc54d bytes) 000360[1] 1775216827.769135 mapwrite 0x48f4d9 thru 0x49b64b (0xc173 bytes) 000361[1] 1775216827.922817 trunc from 0x9df3d8 to 0x2db907 (0x703ad0 bytes) 000363[1] 1775216828.040088 read 0x06f413 thru 0x0782c2 (0x8eb0 bytes) 000365[0] 1775216828.208196 mapwrite 0x1b1cf4 thru 0x1bff8d (0xe29a bytes) 000366[0] 1775216828.503931 read 0x180bd9 thru 0x18b459 (0xa881 bytes) 000371[0] 1775216828.713314 trunc from 0x2db908 to 0x8c7def (0x5ec4e8 bytes) 000372[1] 1775216828.782939 mapread 0x28561e thru 0x28a320 (0x4d03 bytes) 000374[0] 1775216829.019681 read 0x0f8e8c thru 0x100352 (0x74c7 bytes) 000377[1] 1775216829.068013 trunc from 0x8c7df0 to 0x0fa62b (0x7cd7c4 bytes) 000383[1] 1775216829.207095 read 0x0ce137 thru 0x0d0937 (0x2801 bytes) 000384[1] 1775216829.302115 write 0x6100a7 thru 0x614eeb (0x4e45 bytes) 000385[0] 1775216829.306736 mapwrite 0x4a4b7b thru 0x4ac870 (0x7cf6 bytes) 000388[0] 1775216829.634178 mapwrite 0x3c7099 thru 0x3c8d7a (0x1ce2 bytes) 000389[1] 1775216829.808033 mapwrite 0x80cc07 thru 0x817959 (0xad53 bytes) 000391[0] 1775216830.164196 write 0x8658dd thru 0x8758cd (0xfff1 bytes) 000392[0] 1775216830.244270 write 0x8e0b7b thru 0x8eba42 (0xaec8 bytes) 000399[1] 1775216830.536672 write 0x5891ac thru 0x591f6e (0x8dc3 bytes) 000401[0] 1775216830.566176 mapread 0x3efa0b thru 0x3f3ae3 (0x40d9 bytes) 000402[1] 1775216830.613078 mapwrite 0x3ba57c thru 0x3c3b99 (0x961e bytes) 000405[0] 1775216830.910574 write 0x7d0cc4 thru 0x7dafd7 (0xa314 bytes) 000406[1] 1775216831.077131 trunc from 0x8eba43 to 0x71f560 (0x1cc4e2 bytes) 000407[0] 1775216831.168096 read 0x4be5ab thru 0x4c6c60 (0x86b6 bytes) 000410[0] 1775216831.274216 mapwrite 0x6ca6a0 thru 0x6ce086 (0x39e7 bytes) 000411[0] 1775216831.436203 trunc from 0x71f561 to 0x46fe02 (0x2af75e bytes) 000413[0] 1775216831.572170 write 0x17d4d6 thru 0x18b5b4 (0xe0df bytes) 000414[0] 1775216831.639816 trunc from 0x46fe03 to 0x3971e9 (0xd8c19 bytes) 000415[1] 1775216831.701120 read 0x279b42 thru 0x27e609 (0x4ac8 bytes) 000416[1] 1775216831.892041 write 0x1c8952 thru 0x1cda6b (0x511a bytes) 000418[0] 1775216832.000689 write 0x036b00 thru 0x0392e6 (0x27e7 bytes) 000422[1] 1775216832.121659 write 0x6d9060 thru 0x6e5539 (0xc4da bytes) 000424[1] 1775216832.180010 read 0x2c42b8 thru 0x2d01c1 (0xbf0a bytes) 000425[1] 1775216832.220355 read 0x30725f thru 0x314077 (0xce19 bytes) 000428[0] 1775216832.275223 write 0x942742 thru 0x947340 (0x4bff bytes) 000429[1] 1775216832.314293 mapread 0x6b7aa4 thru 0x6c0b65 (0x90c2 bytes) truncating to largest ever: 0x997b5a 000434[0] 1775216832.469817 trunc from 0x947341 to 0x997b59 (0x50819 bytes) 000437[0] 1775216832.599175 mapwrite 0x090467 thru 0x09bede (0xba78 bytes) 000439[1] 1775216832.837061 mapread 0x6ff0de thru 0x703d16 (0x4c39 bytes) 000451[0] 1775216833.102009 write 0x300c27 thru 0x305529 (0x4903 bytes) 000452[0] 1775216833.127832 read 0x61a785 thru 0x6283e0 (0xdc5c bytes) 000454[0] 1775216833.143421 trunc from 0x997b5a to 0x104230 (0x893929 bytes) 000457[1] 1775216833.390782 mapwrite 0x741dfc thru 0x750de1 (0xefe6 bytes) 000458[0] 1775216833.632221 mapwrite 0x8ac87b thru 0x8b0185 (0x390b bytes) 000459[1] 1775216833.782812 write 0x1d5a59 thru 0x1e4380 (0xe928 bytes) 000460[0] 1775216833.831457 mapwrite 0x078e0c thru 0x086736 (0xd92b bytes) 000461[1] 1775216833.959240 mapwrite 0x6dd28f thru 0x6e2077 (0x4de9 bytes) 000462[0] 1775216834.185864 write 0x89c130 thru 0x8a1376 (0x5247 bytes) 000463[1] 1775216834.242650 mapread 0x78a0ab thru 0x793802 (0x9758 bytes) 000466[0] 1775216834.349139 mapwrite 0x2fd16f thru 0x302839 (0x56cb bytes) 000467[1] 1775216834.451215 mapwrite 0x26eb97 thru 0x279d3e (0xb1a8 bytes) 000468[0] 1775216834.604177 write 0x77230a thru 0x780b76 (0xe86d bytes) 000469[0] 1775216834.653234 mapwrite 0x8ac4fe thru 0x8b785d (0xb360 bytes) 000470[1] 1775216834.785986 mapwrite 0x0c5a19 thru 0x0cd68a (0x7c72 bytes) 000472[0] 1775216834.953808 write 0x1d9108 thru 0x1e1e6e (0x8d67 bytes) 000474[0] 1775216835.014963 mapread 0x1ca530 thru 0x1ce44b (0x3f1c bytes) 000476[1] 1775216835.054813 mapread 0x0edcec thru 0x0fbdf2 (0xe107 bytes) 000477[1] 1775216835.119180 mapread 0x0b0296 thru 0x0b6f56 (0x6cc1 bytes) 000482[0] 1775216835.213311 trunc from 0x8b785e to 0x2326b1 (0x6851ac bytes) 000484[1] 1775216835.305218 write 0x1c244c thru 0x1cbcce (0x9883 bytes) 000485[0] 1775216835.372278 write 0x1171f3 thru 0x11f01d (0x7e2b bytes) 000487[1] 1775216835.440492 trunc from 0x2326b2 to 0x092e3a (0x19f877 bytes) 000488[1] 1775216835.504455 trunc from 0x092e3b to 0x9689f2 (0x8d5bb8 bytes) 000491[0] 1775216835.574264 write 0x5df636 thru 0x5e82b9 (0x8c84 bytes) 000492[0] 1775216835.617792 write 0x6819c0 thru 0x687c64 (0x62a5 bytes) 000495[1] 1775216835.722747 mapwrite 0x2aa9b6 thru 0x2ba2da (0xf925 bytes) 000497[1] 1775216835.938331 read 0x1fccc1 thru 0x20c4fb (0xf83b bytes) 000498[0] 1775216835.978194 trunc from 0x9689f3 to 0x724cf9 (0x243cf9 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6836 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x462d13 000006[1] 1775216837.262921 trunc from 00000000 to 0x462d12 (0x462d13 bytes) 000007[0] 1775216837.324763 mapread 0x0cb000 thru 0x0d8ee8 (0xdee9 bytes) 000008[1] 1775216837.377858 write 0x870000 thru 0x87f437 (0xf438 bytes) 000013[1] 1775216837.429827 mapwrite 0x8c6000 thru 0x8d4fe1 (0xefe2 bytes) 000015[1] 1775216837.633124 mapread 0x169000 thru 0x1690fc (0xfd bytes) 000016[0] 1775216837.662932 write_OD 0x106000 thru 0x113fff (0xe000 bytes) truncating to largest ever: 0x4c121d 000018[0] 1775216837.718647 trunc from 0x8d4fe2 to 0x4c121c (0x413dc5 bytes) 000020[1] 1775216837.777943 mapread 0x11f000 thru 0x125b95 (0x6b96 bytes) 000022[1] 1775216837.865737 mapwrite 0x5b0000 thru 0x5b12f8 (0x12f9 bytes) 000023[1] 1775216838.164301 trunc from 0x5b12f9 to 0x0040a8 (0x5ad250 bytes) 000024[1] 1775216838.235378 read 0x001000 thru 0x003fe8 (0x2fe9 bytes) 000026[0] 1775216838.298337 read_OD 0x001000 thru 0x002fff (0x2000 bytes) 000027[0] 1775216838.345620 mapread 0x001000 thru 0x00368a (0x268b bytes) 000030[1] 1775216838.372712 read 0x002000 thru 0x00379f (0x17a0 bytes) 000032[1] 1775216838.387745 mapread 0x003000 thru 0x003d60 (0xd61 bytes) 000035[1] 1775216838.399305 write 0x139000 thru 0x13bc8d (0x2c8e bytes) 000037[0] 1775216838.461927 mapwrite 0x0f3000 thru 0x0f4ebb (0x1ebc bytes) 000039[0] 1775216838.682766 mapread 0x015000 thru 0x01c676 (0x7677 bytes) 000040[1] 1775216838.704788 mapread 0x0c1000 thru 0x0c5416 (0x4417 bytes) 000041[1] 1775216838.790157 write 0x7f0000 thru 0x7fda30 (0xda31 bytes) 000045[0] 1775216838.900203 mapread 0x7ac000 thru 0x7b4cb4 (0x8cb5 bytes) 000046[0] 1775216839.108660 trunc from 0x7fda31 to 0x3cbd35 (0x431cfb bytes) 000050[0] 1775216839.231591 mapwrite 0x59e000 thru 0x5aa5fe (0xc5ff bytes) 000052[0] 1775216839.565045 mapread 0x54d000 thru 0x5545c1 (0x75c2 bytes) 000055[0] 1775216839.682301 mapread 0x19e000 thru 0x1a204b (0x404c bytes) truncating to largest ever: 0x92fbc1 000056[1] 1775216839.834222 trunc from 0x5aa5ff to 0x92fbc0 (0x3855c2 bytes) 000057[0] 1775216839.901106 write_OD 0x8ae000 thru 0x8affff (0x2000 bytes) 000058[0] 1775216839.968035 read_OD 0x496000 thru 0x4a4fff (0xf000 bytes) 000059[0] 1775216839.986034 read_OD 0x469000 thru 0x469fff (0x1000 bytes) truncating to largest ever: 0x9398c0 000060[0] 1775216839.993003 trunc from 0x92fbc1 to 0x9398bf (0x9cff bytes) 000062[1] 1775216840.105860 mapread 0x427000 thru 0x431607 (0xa608 bytes) 000063[0] 1775216840.326089 write_OD 0x2b9000 thru 0x2befff (0x6000 bytes) 000064[0] 1775216840.489120 mapwrite 0x13f000 thru 0x14b248 (0xc249 bytes) 000068[1] 1775216840.663925 read 0x67b000 thru 0x684732 (0x9733 bytes) 000070[1] 1775216840.702364 mapread 0x3e9000 thru 0x3edd0f (0x4d10 bytes) 000073[0] 1775216840.743440 mapread 0x23f000 thru 0x246cef (0x7cf0 bytes) 000074[1] 1775216840.775607 write 0x47a000 thru 0x4882ac (0xe2ad bytes) 000075[1] 1775216840.806075 read 0x499000 thru 0x49aad3 (0x1ad4 bytes) 000076[1] 1775216840.813006 write 0x4bd000 thru 0x4c494e (0x794f bytes) 000079[0] 1775216840.861839 trunc from 0x9398c0 to 0x87a003 (0xbf8bc bytes) 000081[1] 1775216841.028712 read 0x842000 thru 0x84eccd (0xccce bytes) 000086[0] 1775216841.079306 mapread 0x6fe000 thru 0x7014cb (0x34cc bytes) 000087[1] 1775216841.106511 trunc from 0x87a004 to 0x92c979 (0xb2976 bytes) 000088[1] 1775216841.189757 read 0x57a000 thru 0x586d10 (0xcd11 bytes) 000093[1] 1775216841.302110 mapwrite 0x2db000 thru 0x2e73d5 (0xc3d6 bytes) 000095[1] 1775216841.595600 write 0x6b6000 thru 0x6bace9 (0x4cea bytes) 000099[0] 1775216841.640679 mapwrite 0x403000 thru 0x411f91 (0xef92 bytes) 000101[0] 1775216841.919940 write_OD 0x5f8000 thru 0x603fff (0xc000 bytes) 000103[0] 1775216842.017828 mapread 0x081000 thru 0x08884a (0x784b bytes) 000106[0] 1775216842.057415 mapwrite 0x6b8000 thru 0x6b9978 (0x1979 bytes) 000109[0] 1775216842.282600 read_OD 0x2ae000 thru 0x2bbfff (0xe000 bytes) 000110[1] 1775216842.345840 trunc from 0x92c97a to 0x1f1093 (0x73b8e6 bytes) 000114[1] 1775216842.575420 write 0x05f000 thru 0x06d770 (0xe771 bytes) 000115[0] 1775216842.621445 write_OD 0x3fc000 thru 0x3fefff (0x3000 bytes) 000116[1] 1775216842.760447 write 0x3a5000 thru 0x3b25b4 (0xd5b5 bytes) 000119[0] 1775216842.859331 mapread 0x2c2000 thru 0x2c9fe6 (0x7fe7 bytes) 000120[0] 1775216842.986965 mapread 0x2bf000 thru 0x2c3b60 (0x4b61 bytes) 000122[1] 1775216842.988075 write 0x325000 thru 0x32b1c5 (0x61c6 bytes) 000124[0] 1775216843.063407 read_OD 0x05d000 thru 0x069fff (0xd000 bytes) 000127[0] 1775216843.143382 trunc from 0x3ff000 to 0x072a88 (0x38c577 bytes) 000128[1] 1775216843.274449 trunc from 0x072a89 to 0x7f8a5c (0x785fd4 bytes) 000129[0] 1775216843.334603 write_OD 0x057000 thru 0x05bfff (0x5000 bytes) 000130[1] 1775216843.430961 mapread 0x332000 thru 0x341330 (0xf331 bytes) 000132[0] 1775216843.482323 trunc from 0x7f8a5d to 0x4a202b (0x356a31 bytes) 000133[1] 1775216843.556972 mapwrite 0x07f000 thru 0x0849e0 (0x59e1 bytes) 000134[1] 1775216843.819225 mapwrite 0x959000 thru 0x95cf9d (0x3f9e bytes) 000135[1] 1775216844.046108 mapread 0x953000 thru 0x955a8d (0x2a8e bytes) 000137[0] 1775216844.165142 mapread 0x70b000 thru 0x7140be (0x90bf bytes) 000139[1] 1775216844.392649 read 0x16d000 thru 0x170f0a (0x3f0b bytes) 000142[0] 1775216844.401069 read_OD 0x1d8000 thru 0x1dcfff (0x5000 bytes) 000145[1] 1775216844.443267 read 0x09d000 thru 0x0a6668 (0x9669 bytes) 000148[0] 1775216844.468419 mapread 0x49f000 thru 0x4a79d3 (0x89d4 bytes) 000150[1] 1775216844.472770 mapwrite 0x931000 thru 0x93d7b9 (0xc7ba bytes) 000155[1] 1775216844.731009 read 0x5f7000 thru 0x5fbf26 (0x4f27 bytes) 000156[1] 1775216844.771689 mapwrite 0x226000 thru 0x232a17 (0xca18 bytes) 000158[1] 1775216844.937063 read 0x665000 thru 0x66b4e7 (0x64e8 bytes) truncating to largest ever: 0x9b88c1 000159[0] 1775216844.976038 trunc from 0x95cf9e to 0x9b88c0 (0x5b923 bytes) 000160[1] 1775216845.031000 mapread 0x002000 thru 0x00553a (0x353b bytes) 000161[1] 1775216845.075963 mapread 0x653000 thru 0x65f705 (0xc706 bytes) 000164[0] 1775216845.166910 mapwrite 0x71c000 thru 0x71e105 (0x2106 bytes) 000166[0] 1775216845.405955 write_OD 0x2e2000 thru 0x2e6fff (0x5000 bytes) 000167[1] 1775216845.476992 mapwrite 0x16c000 thru 0x16f363 (0x3364 bytes) 000169[1] 1775216845.632564 mapwrite 0x658000 thru 0x65d562 (0x5563 bytes) 000171[1] 1775216845.754828 read 0x67a000 thru 0x688e82 (0xee83 bytes) 000173[0] 1775216845.808336 trunc from 0x9b88c1 to 0x1394e6 (0x87f3da bytes) 000174[1] 1775216845.865229 trunc from 0x1394e7 to 0x3112ba (0x1d7dd4 bytes) 000176[1] 1775216845.939826 write 0x1cd000 thru 0x1cd7ee (0x7ef bytes) 000178[1] 1775216845.953108 mapwrite 0x62b000 thru 0x62e95f (0x3960 bytes) 000180[0] 1775216846.094229 mapread 0x45e000 thru 0x463b25 (0x5b26 bytes) 000181[0] 1775216846.160278 mapread 0x395000 thru 0x39a1be (0x51bf bytes) 000184[1] 1775216846.171261 mapwrite 0x824000 thru 0x82fe49 (0xbe4a bytes) 000185[1] 1775216846.350127 read 0x111000 thru 0x11b2e3 (0xa2e4 bytes) 000186[0] 1775216846.379142 read_OD 0x3d4000 thru 0x3e2fff (0xf000 bytes) 000187[0] 1775216846.427084 trunc from 0x82fe4a to 0x86b364 (0x3b51b bytes) 000188[0] 1775216846.513413 trunc from 0x86b365 to 0x69cd4e (0x1ce616 bytes) 000189[1] 1775216846.545984 trunc from 0x69cd4f to 0x91602a (0x2792dc bytes) 000196[0] 1775216846.721328 write_OD 0x22f000 thru 0x237fff (0x9000 bytes) 000197[0] 1775216846.820900 write_OD 0x331000 thru 0x332fff (0x2000 bytes) 000200[1] 1775216846.890087 mapread 0x7fd000 thru 0x7fe7f0 (0x17f1 bytes) 000206[0] 1775216846.909016 mapread 0x404000 thru 0x413fcd (0xffce bytes) 000208[0] 1775216846.931498 mapwrite 0x2cc000 thru 0x2cd33f (0x1340 bytes) 000210[1] 1775216847.069350 write 0x46f000 thru 0x471194 (0x2195 bytes) 000211[1] 1775216847.113356 trunc from 0x91602b to 0x68044e (0x295bdc bytes) 000212[0] 1775216847.178756 write_OD 0x382000 thru 0x390fff (0xf000 bytes) 000213[1] 1775216847.250065 write 0x99d000 thru 0x9a222e (0x522f bytes) 000215[1] 1775216847.300445 trunc from 0x9a222f to 0x27fa7b (0x7227b3 bytes) 000216[1] 1775216847.381980 write 0x4d2000 thru 0x4d517b (0x317c bytes) 000218[0] 1775216847.459042 trunc from 0x4d517c to 0x02af4f (0x4aa22c bytes) 000219[0] 1775216847.525056 mapwrite 0x6e2000 thru 0x6e8e49 (0x6e4a bytes) 000223[0] 1775216847.799706 read_OD 0x5b3000 thru 0x5b9fff (0x7000 bytes) 000224[0] 1775216847.871327 mapwrite 0x2f5000 thru 0x2f7d64 (0x2d65 bytes) 000226[0] 1775216847.974987 mapread 0x6c0000 thru 0x6cbe91 (0xbe92 bytes) 000228[1] 1775216848.034191 mapread 0x118000 thru 0x127ba6 (0xfba7 bytes) 000229[1] 1775216848.110427 mapread 0x633000 thru 0x641b0b (0xeb0c bytes) 000230[0] 1775216848.152804 write_OD 0x24b000 thru 0x254fff (0xa000 bytes) 000232[0] 1775216848.219557 trunc from 0x6e8e4a to 0x76d026 (0x841dd bytes) 000235[0] 1775216848.318000 read_OD 0x3de000 thru 0x3e5fff (0x8000 bytes) 000241[1] 1775216848.351719 mapread 0x069000 thru 0x06e1f5 (0x51f6 bytes) 000243[0] 1775216848.377301 trunc from 0x76d027 to 0x1da654 (0x5929d2 bytes) 000244[1] 1775216848.449700 mapread 0x020000 thru 0x0244ea (0x44eb bytes) 000248[0] 1775216848.512131 mapwrite 0x276000 thru 0x28293f (0xc940 bytes) 000249[1] 1775216848.734186 mapread 0x047000 thru 0x047f51 (0xf52 bytes) 000250[0] 1775216848.761132 read_OD 0x118000 thru 0x120fff (0x9000 bytes) 000252[1] 1775216848.787869 write 0x2a0000 thru 0x2a6604 (0x6605 bytes) 000254[1] 1775216848.811705 mapwrite 0x120000 thru 0x12a44b (0xa44c bytes) 000255[0] 1775216848.924996 read_OD 0x017000 thru 0x024fff (0xe000 bytes) 000256[1] 1775216848.982865 write 0x4a6000 thru 0x4aec47 (0x8c48 bytes) 000259[0] 1775216849.049068 write_OD 0x8d2000 thru 0x8d2fff (0x1000 bytes) 000260[0] 1775216849.227619 mapread 0x092000 thru 0x0976d5 (0x56d6 bytes) 000263[1] 1775216849.356961 mapwrite 0x200000 thru 0x20d3c6 (0xd3c7 bytes) 000264[0] 1775216849.630548 mapwrite 0x597000 thru 0x5a4aca (0xdacb bytes) 000265[0] 1775216849.940278 read_OD 0x18b000 thru 0x195fff (0xb000 bytes) 000266[0] 1775216849.997042 mapwrite 0x1d9000 thru 0x1daf51 (0x1f52 bytes) 000267[1] 1775216850.222049 read 0x1a8000 thru 0x1b4af3 (0xcaf4 bytes) 000268[1] 1775216850.264212 trunc from 0x8d3000 to 0x97d9a0 (0xaa9a1 bytes) 000269[1] 1775216850.372044 mapread 0x39f000 thru 0x3ad6b1 (0xe6b2 bytes) 000270[0] 1775216850.481504 trunc from 0x97d9a1 to 0x20633f (0x777661 bytes) 000274[0] 1775216850.579689 mapwrite 0x1b7000 thru 0x1bc592 (0x5593 bytes) 000277[0] 1775216850.722504 write_OD 0x773000 thru 0x77afff (0x8000 bytes) 000279[1] 1775216850.764214 mapwrite 0x46d000 thru 0x471f0e (0x4f0f bytes) 000280[0] 1775216850.865496 read_OD 0x0c8000 thru 0x0d6fff (0xf000 bytes) 000281[0] 1775216850.903953 read_OD 0x42a000 thru 0x430fff (0x7000 bytes) 000282[1] 1775216850.923443 mapwrite 0x985000 thru 0x988d0c (0x3d0d bytes) 000285[0] 1775216851.091933 mapwrite 0x1eb000 thru 0x1f7a29 (0xca2a bytes) 000286[0] 1775216851.270502 write_OD 0x36d000 thru 0x370fff (0x4000 bytes) 000287[1] 1775216851.382650 mapread 0x921000 thru 0x922a86 (0x1a87 bytes) 000288[1] 1775216851.448770 mapwrite 0x2c5000 thru 0x2c734c (0x234d bytes) 000291[0] 1775216851.674076 trunc from 0x988d0d to 0x552450 (0x4368bc bytes) 000293[0] 1775216851.740836 read_OD 0x16c000 thru 0x170fff (0x5000 bytes) 000295[1] 1775216851.788681 trunc from 0x552451 to 0x96d841 (0x41b3f1 bytes) 000296[0] 1775216851.855116 mapwrite 0x991000 thru 0x9948f5 (0x38f6 bytes) 000297[0] 1775216851.967708 trunc from 0x9948f6 to 0x86f12b (0x1257ca bytes) 000299[1] 1775216852.014119 mapread 0x160000 thru 0x161659 (0x165a bytes) 000301[1] 1775216852.088006 mapread 0x6e1000 thru 0x6e3527 (0x2528 bytes) 000303[1] 1775216852.132230 read 0x685000 thru 0x68523d (0x23e bytes) 000307[0] 1775216852.166841 mapread 0x815000 thru 0x81cd0e (0x7d0f bytes) 000309[1] 1775216852.197510 mapwrite 0x768000 thru 0x769ded (0x1dee bytes) 000311[1] 1775216852.375096 mapwrite 0x5ec000 thru 0x5f6dab (0xadac bytes) 000312[0] 1775216852.483547 trunc from 0x86f12c to 0x759232 (0x115ef9 bytes) 000315[0] 1775216852.593125 mapwrite 0x356000 thru 0x364fd9 (0xefda bytes) 000316[0] 1775216852.757039 read_OD 0x61f000 thru 0x626fff (0x8000 bytes) 000318[0] 1775216852.850974 write_OD 0x0fb000 thru 0x0fcfff (0x2000 bytes) 000321[1] 1775216852.982457 read 0x2bd000 thru 0x2c7b5b (0xab5c bytes) 000322[1] 1775216853.000635 read 0x072000 thru 0x073d92 (0x1d93 bytes) 000326[0] 1775216853.009421 mapwrite 0x212000 thru 0x21dce4 (0xbce5 bytes) 000327[1] 1775216853.134419 write 0x085000 thru 0x08f84d (0xa84e bytes) 000328[0] 1775216853.162141 write_OD 0x483000 thru 0x48cfff (0xa000 bytes) 000329[1] 1775216853.261447 mapread 0x136000 thru 0x13ee31 (0x8e32 bytes) 000330[1] 1775216853.303021 mapwrite 0x90a000 thru 0x91229b (0x829c bytes) 000331[0] 1775216853.507861 trunc from 0x91229c to 0x2201e9 (0x6f20b2 bytes) 000332[1] 1775216853.564052 mapwrite 0x438000 thru 0x4404ff (0x8500 bytes) 000333[1] 1775216853.734361 mapread 0x06c000 thru 0x0758ec (0x98ed bytes) 000335[0] 1775216853.823809 mapwrite 0x547000 thru 0x54855b (0x155c bytes) 000336[0] 1775216853.956101 mapread 0x3f4000 thru 0x3ff207 (0xb208 bytes) 000338[0] 1775216854.014614 mapread 0x3b2000 thru 0x3bda11 (0xba12 bytes) 000339[1] 1775216854.046866 mapread 0x1e8000 thru 0x1f46cb (0xc6cc bytes) 000340[0] 1775216854.117761 mapwrite 0x6d9000 thru 0x6de856 (0x5857 bytes) 000343[0] 1775216854.414215 trunc from 0x6de857 to 0x365a9c (0x378dba bytes) 000345[1] 1775216854.538074 read 0x282000 thru 0x28b850 (0x9851 bytes) 000346[0] 1775216854.718456 mapwrite 0x0a7000 thru 0x0b2c92 (0xbc93 bytes) 000347[0] 1775216854.919579 mapwrite 0x302000 thru 0x30a972 (0x8973 bytes) 000350[1] 1775216855.078525 trunc from 0x365a9d to 0x838c36 (0x4d319a bytes) 000351[1] 1775216855.161043 write 0x13d000 thru 0x141f3b (0x4f3c bytes) 000352[1] 1775216855.205019 write 0x267000 thru 0x272818 (0xb819 bytes) 000353[1] 1775216855.237672 write 0x41e000 thru 0x4230ee (0x50ef bytes) 000357[1] 1775216855.336155 read 0x4ee000 thru 0x4f664f (0x8650 bytes) 000359[1] 1775216855.364337 read 0x26a000 thru 0x26a795 (0x796 bytes) 000361[1] 1775216855.410656 write 0x3d9000 thru 0x3dfed5 (0x6ed6 bytes) 000365[0] 1775216855.499163 trunc from 0x838c37 to 0x8c7f6a (0x8f334 bytes) 000366[1] 1775216855.553401 mapwrite 0x366000 thru 0x36c298 (0x6299 bytes) 000367[1] 1775216855.669912 read 0x254000 thru 0x25681c (0x281d bytes) 000368[1] 1775216855.681633 read 0x88b000 thru 0x88f912 (0x4913 bytes) 000369[0] 1775216855.708878 mapwrite 0x195000 thru 0x196dd3 (0x1dd4 bytes) 000371[0] 1775216855.852858 trunc from 0x8c7f6b to 0x638af4 (0x28f476 bytes) 000374[1] 1775216855.905140 write 0x9bd000 thru 0x9c4cd2 (0x7cd3 bytes) 000381[1] 1775216856.027551 trunc from 0x9c4cd3 to 0x692298 (0x332a3a bytes) 000382[1] 1775216856.061680 read 0x071000 thru 0x079604 (0x8605 bytes) 000383[0] 1775216856.097715 mapread 0x18d000 thru 0x18fc5d (0x2c5e bytes) 000384[1] 1775216856.150582 trunc from 0x692299 to 0x009e80 (0x688418 bytes) 000385[0] 1775216856.191874 trunc from 0x009e81 to 0x95188f (0x947a0f bytes) 000388[1] 1775216856.275855 mapwrite 0x56d000 thru 0x572088 (0x5089 bytes) 000389[1] 1775216856.481212 mapread 0x7c4000 thru 0x7cd653 (0x9654 bytes) 000390[1] 1775216856.519666 mapwrite 0x1d8000 thru 0x1decab (0x6cac bytes) 000392[1] 1775216856.611534 mapwrite 0x89b000 thru 0x89d7eb (0x27ec bytes) 000393[1] 1775216856.713426 mapread 0x306000 thru 0x30c69e (0x669f bytes) 000394[1] 1775216856.753039 read 0x470000 thru 0x4791e7 (0x91e8 bytes) 000396[0] 1775216856.797414 mapwrite 0x84e000 thru 0x857035 (0x9036 bytes) 000398[1] 1775216856.919070 trunc from 0x951890 to 0x0ffdd1 (0x851abe bytes) 000399[1] 1775216856.955066 mapread 0x054000 thru 0x05810a (0x410b bytes) 000402[0] 1775216857.008856 read_OD 0x005000 thru 0x00ffff (0xb000 bytes) 000403[1] 1775216857.043318 write 0x77b000 thru 0x788df8 (0xddf9 bytes) 000407[1] 1775216857.157436 mapread 0x625000 thru 0x625799 (0x79a bytes) 000410[0] 1775216857.198205 trunc from 0x788df9 to 0x72b308 (0x5daf0 bytes) 000412[1] 1775216857.289243 trunc from 0x72b309 to 0x771f83 (0x46c7b bytes) 000414[1] 1775216857.348990 mapread 0x756000 thru 0x764ecb (0xeecc bytes) 000415[1] 1775216857.416447 mapread 0x6ff000 thru 0x703fea (0x4feb bytes) 000416[1] 1775216857.432729 trunc from 0x771f84 to 0x1b4a95 (0x5bd4ee bytes) 000419[0] 1775216857.457848 mapwrite 0x0fc000 thru 0x0fe8ae (0x28af bytes) 000425[0] 1775216857.619648 trunc from 0x1b4a96 to 0x56b3e3 (0x3b694e bytes) 000426[0] 1775216857.669372 trunc from 0x56b3e4 to 0x0278e0 (0x543b03 bytes) 000427[0] 1775216857.697242 trunc from 0x0278e1 to 0x1c5b59 (0x19e279 bytes) 000437[0] 1775216857.821943 write_OD 0x009000 thru 0x012fff (0xa000 bytes) 000440[1] 1775216857.891593 mapread 0x0dc000 thru 0x0e6ae0 (0xaae1 bytes) 000444[1] 1775216857.931865 write 0x28f000 thru 0x28fb4b (0xb4c bytes) 000445[1] 1775216857.965822 read 0x259000 thru 0x265414 (0xc415 bytes) 000447[1] 1775216858.020549 write 0x1ea000 thru 0x1f3d28 (0x9d29 bytes) 000449[1] 1775216858.022126 mapwrite 0x93f000 thru 0x947c12 (0x8c13 bytes) 000452[1] 1775216858.254638 write 0x1b4000 thru 0x1c11d4 (0xd1d5 bytes) 000453[1] 1775216858.255975 read 0x677000 thru 0x6860aa (0xf0ab bytes) 000455[1] 1775216858.279815 write 0x658000 thru 0x65d06d (0x506e bytes) 000460[0] 1775216858.401316 mapread 0x2cf000 thru 0x2d50e7 (0x60e8 bytes) 000461[0] 1775216858.581676 write_OD 0x5e5000 thru 0x5ecfff (0x8000 bytes) 000465[1] 1775216858.704293 mapwrite 0x9ca000 thru 0x9ca75c (0x75d bytes) 000466[1] 1775216858.927369 mapwrite 0x395000 thru 0x397c47 (0x2c48 bytes) 000470[0] 1775216859.368338 mapread 0x9bb000 thru 0x9c06a7 (0x56a8 bytes) 000472[1] 1775216859.511331 trunc from 0x9ca75d to 0x978e09 (0x51953 bytes) 000473[0] 1775216859.618771 mapread 0x6e7000 thru 0x6f1270 (0xa271 bytes) 000474[0] 1775216859.769725 read_OD 0x78c000 thru 0x791fff (0x6000 bytes) 000475[0] 1775216859.817181 mapread 0x494000 thru 0x49bbc6 (0x7bc7 bytes) 000477[1] 1775216859.861191 mapwrite 0x79f000 thru 0x7a9954 (0xa955 bytes) 000478[0] 1775216860.101274 write_OD 0x3a8000 thru 0x3b4fff (0xd000 bytes) 000480[0] 1775216860.230858 write_OD 0x866000 thru 0x86bfff (0x6000 bytes) 000481[1] 1775216860.351124 mapread 0x036000 thru 0x044304 (0xe305 bytes) 000482[1] 1775216860.397968 mapwrite 0x246000 thru 0x254e0b (0xee0c bytes) 000485[0] 1775216860.555680 mapwrite 0x024000 thru 0x0286af (0x46b0 bytes) 000491[1] 1775216860.874634 trunc from 0x978e0a to 0x427cee (0x55111b bytes) 000492[0] 1775216860.946957 mapwrite 0x249000 thru 0x24f3e2 (0x63e3 bytes) 000493[0] 1775216861.060980 mapwrite 0x019000 thru 0x0197bb (0x7bc bytes) 000494[1] 1775216861.138321 read 0x2f8000 thru 0x2fd707 (0x5708 bytes) 000497[0] 1775216861.179547 mapread 0x114000 thru 0x11f98b (0xb98c bytes) 000498[0] 1775216861.270609 read_OD 0x16c000 thru 0x16efff (0x3000 bytes) 000500[1] 1775216861.279073 read 0x11b000 thru 0x12736a (0xc36b bytes) All operations completed A-OK! PASS 16a (65s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 07:47:47 (1775216867) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.475263 s, 17.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.468785 s, 17.9 MB/s Chance of close/open is 1 in 50 Seed set to 6870 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1775216871.632853 mapwrite 0x96aeca thru 0x96b97c (0xab3 bytes) 000006[1] 1775216871.888074 mapwrite 0x5abc5d thru 0x5ad1e7 (0x158b bytes) truncating to largest ever: 0x3d5611 000008[1] 1775216872.093108 trunc from 0x96b97d to 0x3d5610 (0x59636c bytes) 000010[0] 1775216872.199757 mapread 0x09f92d thru 0x09f93e (0x12 bytes) 000011[1] 1775216872.247457 trunc from 0x3d5611 to 0x0bbe8f (0x319781 bytes) 000013[0] 1775216872.328842 mapwrite 0x22e3b8 thru 0x22eb8a (0x7d3 bytes) 000014[1] 1775216872.472402 mapwrite 0x95c33d thru 0x95cb37 (0x7fb bytes) 000015[0] 1775216872.606367 read 0x1d9ba3 thru 0x1db170 (0x15ce bytes) 000016[0] 1775216872.638614 read 0x6e5c47 thru 0x6e66f2 (0xaac bytes) 000021[1] 1775216872.680567 write 0x618534 thru 0x619cdf (0x17ac bytes) truncating to largest ever: 0x578e91 000023[1] 1775216872.728236 trunc from 0x95cb38 to 0x578e90 (0x3e3ca7 bytes) 000024[0] 1775216872.787649 mapwrite 0x2ec8b1 thru 0x2ee7b3 (0x1f03 bytes) 000025[0] 1775216873.044901 read 0x020320 thru 0x021cb5 (0x1996 bytes) 000026[0] 1775216873.126135 mapwrite 0x67d82e thru 0x67e65f (0xe32 bytes) 000028[0] 1775216873.437885 write 0x122ad4 thru 0x123df9 (0x1326 bytes) 000029[0] 1775216873.491419 mapread 0x37c8a5 thru 0x37ca79 (0x1d5 bytes) 000032[1] 1775216873.536399 mapwrite 0x4b3a80 thru 0x4b426f (0x7f0 bytes) 000033[1] 1775216873.663880 read 0x29725b thru 0x298445 (0x11eb bytes) 000035[1] 1775216873.718194 trunc from 0x67e660 to 0x539cfd (0x144962 bytes) 000036[0] 1775216873.783556 write 0x1ca1dc thru 0x1ca222 (0x47 bytes) 000037[1] 1775216873.835338 mapwrite 0x498778 thru 0x49a5f4 (0x1e7d bytes) 000041[0] 1775216874.168301 trunc from 0x539cfe to 0x100622 (0x4396db bytes) 000042[1] 1775216874.207339 read 0x053cd9 thru 0x054f0a (0x1232 bytes) 000047[0] 1775216874.306844 read 0x01b321 thru 0x01c109 (0xde9 bytes) 000048[0] 1775216874.349719 mapwrite 0x8748a6 thru 0x876030 (0x178b bytes) 000049[1] 1775216874.697122 read 0x6cc0d6 thru 0x6cccf1 (0xc1c bytes) 000050[1] 1775216874.759266 mapread 0x446e57 thru 0x448851 (0x19fb bytes) 000053[1] 1775216874.825877 read 0x0f6171 thru 0x0f7b6d (0x19fd bytes) 000054[0] 1775216874.854281 read 0x7d5a95 thru 0x7d63b9 (0x925 bytes) 000056[1] 1775216874.870287 read 0x6e492a thru 0x6e6075 (0x174c bytes) 000057[0] 1775216874.878035 read 0x044bd0 thru 0x045aec (0xf1d bytes) 000058[0] 1775216874.884537 read 0x69ee6c thru 0x69f935 (0xaca bytes) 000060[0] 1775216874.896060 mapread 0x2b474a thru 0x2b4c68 (0x51f bytes) 000062[1] 1775216874.911310 mapread 0x25b5ac thru 0x25cd0a (0x175f bytes) 000063[1] 1775216874.937004 mapread 0x7851ea thru 0x785fdc (0xdf3 bytes) 000065[1] 1775216874.949184 write 0x58df68 thru 0x58f383 (0x141c bytes) 000066[0] 1775216875.001880 trunc from 0x876031 to 0x52e415 (0x347c1b bytes) 000067[1] 1775216875.160540 mapwrite 0x1e3773 thru 0x1e4840 (0x10ce bytes) 000069[1] 1775216875.314311 mapwrite 0x2ef813 thru 0x2f11db (0x19c9 bytes) 000070[1] 1775216875.402006 mapwrite 0x984f43 thru 0x9854c8 (0x586 bytes) 000072[0] 1775216875.501240 mapwrite 0x08b6e5 thru 0x08cafa (0x1416 bytes) 000074[0] 1775216875.612807 mapread 0x3129b3 thru 0x3132ce (0x91c bytes) 000075[1] 1775216875.649873 read 0x3addac thru 0x3af510 (0x1765 bytes) 000076[1] 1775216875.687312 mapread 0x2e6df8 thru 0x2e8b24 (0x1d2d bytes) 000078[1] 1775216875.734453 read 0x4a62e6 thru 0x4a6d45 (0xa60 bytes) 000079[0] 1775216875.761411 read 0x5daf41 thru 0x5dc498 (0x1558 bytes) 000080[0] 1775216875.779891 mapwrite 0x2dd5f8 thru 0x2dd771 (0x17a bytes) 000081[1] 1775216875.973310 read 0x3c8fda thru 0x3ca789 (0x17b0 bytes) 000082[0] 1775216876.006799 mapread 0x3db713 thru 0x3dc3b8 (0xca6 bytes) 000083[0] 1775216876.032779 read 0x2b7e63 thru 0x2b91c4 (0x1362 bytes) 000084[1] 1775216876.042510 write 0x05edc8 thru 0x05f767 (0x9a0 bytes) 000092[0] 1775216876.135306 mapwrite 0x3b6656 thru 0x3b6ff5 (0x9a0 bytes) 000093[1] 1775216876.196562 write 0x92e455 thru 0x92fce6 (0x1892 bytes) 000095[1] 1775216876.235020 mapread 0x734b95 thru 0x735528 (0x994 bytes) 000098[0] 1775216876.275938 read 0x08d415 thru 0x08f336 (0x1f22 bytes) 000099[1] 1775216876.307771 write 0x065a25 thru 0x065e62 (0x43e bytes) truncating to largest ever: 0x69ada3 000100[1] 1775216876.341505 trunc from 0x9854c9 to 0x69ada2 (0x2ea726 bytes) 000107[1] 1775216876.476647 mapread 0x10266b thru 0x103932 (0x12c8 bytes) 000108[1] 1775216876.505630 trunc from 0x69ada3 to 0x54ebc4 (0x14c1de bytes) 000110[0] 1775216876.531420 write 0x4ae8f5 thru 0x4affe4 (0x16f0 bytes) 000115[1] 1775216876.628499 read 0x1d080f thru 0x1d1e82 (0x1674 bytes) 000116[0] 1775216876.649961 write 0x42aaa3 thru 0x42c9cc (0x1f2a bytes) 000117[1] 1775216876.675171 read 0x2042c7 thru 0x205429 (0x1163 bytes) truncating to largest ever: 0x9d973f 000121[1] 1775216876.749174 trunc from 0x54ebc5 to 0x9d973e (0x48ab7a bytes) 000124[1] 1775216876.799947 mapread 0x9a9e03 thru 0x9abcb8 (0x1eb6 bytes) 000125[1] 1775216876.829281 mapwrite 0x19de24 thru 0x19f3f5 (0x15d2 bytes) 000126[0] 1775216876.903909 mapwrite 0x4522ef thru 0x452476 (0x188 bytes) 000129[0] 1775216877.023791 write 0x735253 thru 0x7356ce (0x47c bytes) 000130[0] 1775216877.045106 read 0x39bfa1 thru 0x39da02 (0x1a62 bytes) 000132[0] 1775216877.107634 mapwrite 0x55b96b thru 0x55b971 (0x7 bytes) 000133[1] 1775216877.183454 mapread 0x0fe0b3 thru 0x0ffb4c (0x1a9a bytes) 000136[0] 1775216877.245917 read 0x6c7fd2 thru 0x6c8de8 (0xe17 bytes) 000139[0] 1775216877.324382 write 0x53dcd3 thru 0x53ebc9 (0xef7 bytes) 000140[0] 1775216877.347592 write 0x098f4c thru 0x09a6cd (0x1782 bytes) 000141[0] 1775216877.421227 write 0x6ab85c thru 0x6abf61 (0x706 bytes) 000142[0] 1775216877.432058 mapread 0x3ce78e thru 0x3ce796 (0x9 bytes) 000149[0] 1775216877.496152 mapread 0x4d2f05 thru 0x4d3b94 (0xc90 bytes) 000151[1] 1775216877.535921 mapread 0x82dba0 thru 0x82e9b1 (0xe12 bytes) 000154[1] 1775216877.611263 write 0x8d3d43 thru 0x8d5158 (0x1416 bytes) 000155[1] 1775216877.661653 write 0x6f99a0 thru 0x6f9c69 (0x2ca bytes) 000158[0] 1775216877.689047 trunc from 0x9d973f to 0x5c6156 (0x4135e8 bytes) 000159[1] 1775216877.792318 mapwrite 0x9cdbe6 thru 0x9cf2c6 (0x16e1 bytes) 000162[1] 1775216878.014260 read 0x6627a7 thru 0x663d3e (0x1598 bytes) 000164[0] 1775216878.030374 read 0x2d5866 thru 0x2d5dc2 (0x55d bytes) 000166[0] 1775216878.156533 mapwrite 0x122a16 thru 0x124740 (0x1d2b bytes) 000168[1] 1775216878.511434 write 0x9a9646 thru 0x9a99d4 (0x38f bytes) 000169[0] 1775216878.582910 read 0x7f7d9a thru 0x7f7e71 (0xd8 bytes) truncating to largest ever: 0x9fe6e7 000170[0] 1775216878.629901 trunc from 0x9cf2c7 to 0x9fe6e6 (0x2f420 bytes) 000172[0] 1775216878.814616 mapread 0x39d504 thru 0x39f260 (0x1d5d bytes) 000173[0] 1775216878.850696 read 0x1cc885 thru 0x1ccb0a (0x286 bytes) 000175[1] 1775216878.875275 read 0x0d21ec thru 0x0d3854 (0x1669 bytes) 000176[1] 1775216878.926619 trunc from 0x9fe6e7 to 0x907911 (0xf6dd5 bytes) 000177[1] 1775216879.054054 mapwrite 0x6fa9cb thru 0x6fab64 (0x19a bytes) 000181[1] 1775216879.322580 mapwrite 0x1aac1b thru 0x1ab7d9 (0xbbf bytes) 000184[1] 1775216879.585066 write 0x420823 thru 0x420d58 (0x536 bytes) 000186[1] 1775216879.687105 read 0x6be95d thru 0x6bfcc3 (0x1367 bytes) 000191[1] 1775216879.830314 mapread 0x4d9c60 thru 0x4d9d5c (0xfd bytes) 000192[0] 1775216879.894702 read 0x8098e9 thru 0x80b59c (0x1cb4 bytes) 000193[1] 1775216879.935885 trunc from 0x907912 to 0x947478 (0x3fb67 bytes) 000194[0] 1775216880.007663 read 0x775a18 thru 0x776861 (0xe4a bytes) 000196[1] 1775216880.120681 write 0x309d70 thru 0x309ddf (0x70 bytes) 000198[1] 1775216880.168522 mapwrite 0x10b99a thru 0x10c931 (0xf98 bytes) 000203[1] 1775216880.313809 write 0x535ac7 thru 0x537151 (0x168b bytes) 000204[1] 1775216880.341634 read 0x2bf05f thru 0x2c0d0e (0x1cb0 bytes) 000209[1] 1775216880.402704 mapread 0x57bb7a thru 0x57c762 (0xbe9 bytes) 000213[1] 1775216880.447953 mapwrite 0x0ee373 thru 0x0eeda0 (0xa2e bytes) 000214[0] 1775216880.629955 read 0x83a9af thru 0x83c573 (0x1bc5 bytes) 000216[0] 1775216880.668042 trunc from 0x947479 to 0x1cb134 (0x77c344 bytes) 000217[0] 1775216880.740444 write 0x6b853a thru 0x6b8ea8 (0x96f bytes) 000219[1] 1775216880.745098 mapwrite 0x56778e thru 0x56922b (0x1a9e bytes) 000220[1] 1775216880.900527 write 0x000fbf thru 0x002d99 (0x1ddb bytes) 000221[1] 1775216880.926800 mapwrite 0x1ca25f thru 0x1cbc0d (0x19af bytes) 000222[0] 1775216880.979788 read 0x02f4cb thru 0x02fa76 (0x5ac bytes) 000223[0] 1775216881.010854 read 0x49183a thru 0x4931ea (0x19b1 bytes) 000224[0] 1775216881.040425 mapwrite 0x137452 thru 0x138e4f (0x19fe bytes) 000225[0] 1775216881.156985 mapwrite 0x3181a1 thru 0x31a039 (0x1e99 bytes) 000228[0] 1775216881.234635 trunc from 0x6b8ea9 to 0x389361 (0x32fb47 bytes) 000230[1] 1775216881.270404 read 0x2ea771 thru 0x2eab3f (0x3cf bytes) 000231[1] 1775216881.311331 mapwrite 0x570317 thru 0x570e1c (0xb06 bytes) 000232[1] 1775216881.404167 mapread 0x371c2e thru 0x372d8d (0x1160 bytes) 000234[1] 1775216881.419899 write 0x726ad6 thru 0x726b96 (0xc1 bytes) 000237[1] 1775216881.451250 mapwrite 0x32c2b6 thru 0x32d9bd (0x1708 bytes) 000241[1] 1775216881.574123 mapwrite 0x658333 thru 0x65a32c (0x1ffa bytes) 000245[0] 1775216881.762077 mapwrite 0x09f52d thru 0x0a0ebb (0x198f bytes) 000246[0] 1775216881.899284 write 0x836490 thru 0x837408 (0xf79 bytes) 000248[1] 1775216881.944355 read 0x57e82f thru 0x57fce1 (0x14b3 bytes) 000249[1] 1775216881.973702 read 0x277c2c thru 0x278327 (0x6fc bytes) 000250[0] 1775216882.001091 mapread 0x260bf2 thru 0x262223 (0x1632 bytes) 000251[0] 1775216882.078352 mapwrite 0x8195c1 thru 0x81af7d (0x19bd bytes) 000252[1] 1775216882.260971 write 0x365f3d thru 0x367cab (0x1d6f bytes) 000253[0] 1775216882.323288 read 0x0082a6 thru 0x008974 (0x6cf bytes) 000255[1] 1775216882.398554 mapread 0x680c67 thru 0x6823e0 (0x177a bytes) 000256[0] 1775216882.478277 mapwrite 0x3ef91b thru 0x3f17b9 (0x1e9f bytes) 000257[1] 1775216882.712817 mapread 0x57763d thru 0x578d5e (0x1722 bytes) 000259[1] 1775216882.780216 mapread 0x638090 thru 0x639f51 (0x1ec2 bytes) 000260[1] 1775216882.820442 read 0x415dbe thru 0x4169ca (0xc0d bytes) 000264[1] 1775216882.955094 write 0x3b0da3 thru 0x3b1991 (0xbef bytes) 000267[0] 1775216883.035029 read 0x0c2a68 thru 0x0c461d (0x1bb6 bytes) 000271[1] 1775216883.136130 read 0x1bf26b thru 0x1c0357 (0x10ed bytes) 000273[0] 1775216883.166515 mapread 0x2ed2e7 thru 0x2ed37d (0x97 bytes) 000274[0] 1775216883.203351 write 0x56dca9 thru 0x56f08b (0x13e3 bytes) 000276[0] 1775216883.270278 read 0x708de0 thru 0x708f9d (0x1be bytes) 000278[1] 1775216883.296369 mapread 0x107571 thru 0x1077ee (0x27e bytes) 000280[0] 1775216883.312949 trunc from 0x837409 to 0x7b9942 (0x7dac6 bytes) 000281[1] 1775216883.360973 mapread 0x28b725 thru 0x28d226 (0x1b02 bytes) 000283[0] 1775216883.462962 mapread 0x692c44 thru 0x694545 (0x1902 bytes) 000284[0] 1775216883.492542 mapwrite 0x520715 thru 0x521188 (0xa74 bytes) 000285[1] 1775216883.558997 read 0x40e5b9 thru 0x40ec88 (0x6d0 bytes) 000286[0] 1775216883.638193 trunc from 0x7b9943 to 0x53445d (0x2854e5 bytes) 000287[1] 1775216883.800871 read 0x475c75 thru 0x476902 (0xc8e bytes) 000288[1] 1775216883.872102 trunc from 0x53445e to 0x49a142 (0x9a31b bytes) 000290[1] 1775216884.037486 read 0x22553e thru 0x226b6c (0x162f bytes) 000291[1] 1775216884.084101 write 0x6f102e thru 0x6f29ba (0x198d bytes) 000292[0] 1775216884.113145 mapread 0x24072e thru 0x240f92 (0x865 bytes) 000297[1] 1775216884.160875 write 0x79a3d5 thru 0x79c374 (0x1fa0 bytes) 000299[0] 1775216884.162713 mapread 0x4cf307 thru 0x4cf843 (0x53d bytes) 000302[0] 1775216884.206810 read 0x306321 thru 0x307d88 (0x1a68 bytes) 000303[1] 1775216884.216883 mapread 0x219d82 thru 0x21ba0e (0x1c8d bytes) 000305[1] 1775216884.238801 trunc from 0x79c375 to 0x75503d (0x47337 bytes) 000306[0] 1775216884.290688 read 0x0eb139 thru 0x0ecfbc (0x1e84 bytes) 000307[1] 1775216884.313934 trunc from 0x75503e to 0x3c79db (0x38d662 bytes) 000309[1] 1775216884.343921 read 0x365c3f thru 0x366138 (0x4fa bytes) 000311[1] 1775216884.419922 read 0x3598d5 thru 0x35b02d (0x1759 bytes) 000312[1] 1775216884.420962 trunc from 0x3c79dc to 0x2aa111 (0x11d8ca bytes) 000313[1] 1775216884.465076 read 0x1bdda4 thru 0x1be80f (0xa6c bytes) 000314[1] 1775216884.469199 read 0x24636f thru 0x246481 (0x113 bytes) 000320[1] 1775216884.522345 mapwrite 0x7bb5f5 thru 0x7bb7af (0x1bb bytes) 000321[1] 1775216884.703781 read 0x6db0fc thru 0x6dcc4f (0x1b54 bytes) 000322[1] 1775216884.736151 write 0x1a5a3f thru 0x1a69f3 (0xfb5 bytes) 000324[0] 1775216884.782550 trunc from 0x7bb7b0 to 0x6dc0bd (0xdf6f2 bytes) 000326[0] 1775216884.939011 read 0x57842d thru 0x579d55 (0x1929 bytes) 000327[1] 1775216884.944945 trunc from 0x6dc0be to 0x8cb9ea (0x1ef92d bytes) 000329[1] 1775216885.039452 mapwrite 0x46dc10 thru 0x46f95e (0x1d4f bytes) 000333[1] 1775216885.334608 read 0x182b9a thru 0x1837a5 (0xc0c bytes) 000334[0] 1775216885.384429 write 0x636dd1 thru 0x6373f7 (0x627 bytes) 000335[0] 1775216885.473872 mapwrite 0x3038f1 thru 0x304500 (0xc10 bytes) 000337[1] 1775216885.639111 trunc from 0x8cb9eb to 0x9f22a1 (0x1268b7 bytes) 000340[0] 1775216885.834000 write 0x51a039 thru 0x51a71b (0x6e3 bytes) 000342[0] 1775216885.981194 mapwrite 0x6229e0 thru 0x623136 (0x757 bytes) 000343[0] 1775216886.191283 mapread 0x193bac thru 0x193d49 (0x19e bytes) 000346[0] 1775216886.296639 read 0x43bb91 thru 0x43d92a (0x1d9a bytes) 000347[1] 1775216886.305183 mapwrite 0x39dd1a thru 0x39de44 (0x12b bytes) 000349[0] 1775216886.450151 write 0x50fdac thru 0x51194a (0x1b9f bytes) 000353[1] 1775216886.552296 mapread 0x74b8a1 thru 0x74bc8c (0x3ec bytes) 000354[1] 1775216886.622843 mapwrite 0x9961ce thru 0x996eae (0xce1 bytes) 000356[1] 1775216886.857222 read 0x4ff4c3 thru 0x4ff89a (0x3d8 bytes) 000357[1] 1775216886.911740 write 0x05ba27 thru 0x05bee7 (0x4c1 bytes) 000359[1] 1775216886.957702 mapread 0x4d5714 thru 0x4d5d24 (0x611 bytes) 000364[1] 1775216887.004012 trunc from 0x9f22a2 to 0x408b89 (0x5e9718 bytes) 000365[0] 1775216887.082156 mapwrite 0x37c041 thru 0x37cc35 (0xbf5 bytes) 000366[0] 1775216887.196790 write 0x3369e3 thru 0x336c54 (0x272 bytes) 000367[1] 1775216887.227093 write 0x82e9bc thru 0x82f974 (0xfb9 bytes) 000370[0] 1775216887.315567 write 0x15fc34 thru 0x160979 (0xd46 bytes) 000373[1] 1775216887.427830 read 0x2da2de thru 0x2db8da (0x15fd bytes) 000374[0] 1775216887.506414 write 0x9d04e2 thru 0x9d06ca (0x1e9 bytes) 000376[0] 1775216887.607875 trunc from 0x9d06cb to 0x456260 (0x57a46a bytes) 000377[1] 1775216887.685498 trunc from 0x456261 to 0x19548a (0x2c0dd6 bytes) 000378[1] 1775216887.810406 mapwrite 0x80c272 thru 0x80ca77 (0x806 bytes) 000380[0] 1775216888.036739 mapread 0x6c3b7b thru 0x6c4f1d (0x13a3 bytes) 000382[1] 1775216888.126574 write 0x099b71 thru 0x09acc8 (0x1158 bytes) 000383[0] 1775216888.158006 trunc from 0x80ca78 to 0x0687f9 (0x7a427e bytes) 000384[0] 1775216888.260458 mapread 0x029588 thru 0x02a68a (0x1103 bytes) 000389[0] 1775216888.444563 read 0x01ddcf thru 0x01e197 (0x3c9 bytes) 000391[0] 1775216888.498966 mapread 0x02acd3 thru 0x02c6a6 (0x19d4 bytes) 000392[0] 1775216888.530421 mapwrite 0x780fe3 thru 0x7814dd (0x4fb bytes) 000394[0] 1775216888.759356 mapread 0x5b7ce8 thru 0x5b88c3 (0xbdc bytes) 000396[1] 1775216888.785194 trunc from 0x7814de to 0x7fa1a6 (0x78cc9 bytes) 000398[0] 1775216888.952626 mapwrite 0x6816af thru 0x681b24 (0x476 bytes) 000400[1] 1775216889.119010 trunc from 0x7fa1a7 to 0x77ba65 (0x7e741 bytes) 000401[1] 1775216889.173004 mapread 0x0de316 thru 0x0e00c6 (0x1db1 bytes) 000415[0] 1775216889.405078 write 0x9ed26c thru 0x9ed7d0 (0x565 bytes) 000417[0] 1775216889.463996 read 0x2afe33 thru 0x2b05eb (0x7b9 bytes) 000418[1] 1775216889.503132 mapread 0x5367f2 thru 0x53836d (0x1b7c bytes) 000421[0] 1775216889.658394 mapread 0x45c47f thru 0x45d541 (0x10c3 bytes) 000422[1] 1775216889.775620 trunc from 0x9ed7d1 to 0x36e75b (0x67f075 bytes) 000424[1] 1775216889.856228 read 0x32b3af thru 0x32ca68 (0x16ba bytes) 000427[1] 1775216889.965789 read 0x270127 thru 0x271ade (0x19b8 bytes) 000428[1] 1775216889.966390 mapwrite 0x95cb80 thru 0x95cde8 (0x269 bytes) 000429[1] 1775216890.100720 write 0x327648 thru 0x327bca (0x583 bytes) 000430[1] 1775216890.147911 read 0x3c5751 thru 0x3c6dc3 (0x1673 bytes) 000432[1] 1775216890.198188 trunc from 0x95cde9 to 0x690dca (0x2cc01e bytes) 000435[0] 1775216890.232010 mapread 0x133703 thru 0x133d7d (0x67b bytes) 000437[0] 1775216890.333786 write 0x60f440 thru 0x6104ff (0x10c0 bytes) 000439[0] 1775216890.427165 read 0x40dfe3 thru 0x40fd1b (0x1d39 bytes) 000442[1] 1775216890.489432 write 0x3ba65a thru 0x3bb8d7 (0x127e bytes) 000443[1] 1775216890.557849 read 0x0a9d5a thru 0x0ab03a (0x12e1 bytes) 000448[0] 1775216890.682068 read 0x35ed15 thru 0x360c4f (0x1f3b bytes) 000451[1] 1775216890.747697 trunc from 0x690dcb to 0x3f1d73 (0x29f057 bytes) 000452[1] 1775216890.804208 read 0x196b1b thru 0x19714a (0x630 bytes) 000454[0] 1775216890.905404 write 0x48b0b5 thru 0x48c4b0 (0x13fc bytes) 000455[0] 1775216890.939653 trunc from 0x48c4b1 to 0x5af09a (0x122bea bytes) 000459[1] 1775216891.017874 mapwrite 0x7203e2 thru 0x72151b (0x113a bytes) 000460[1] 1775216891.267265 trunc from 0x72151c to 0x7cec6c (0xad751 bytes) 000463[0] 1775216891.358121 write 0x527eed thru 0x529c08 (0x1d1c bytes) 000466[0] 1775216891.430407 read 0x5daf3a thru 0x5dc2a5 (0x136c bytes) 000467[1] 1775216891.451700 read 0x568be3 thru 0x56a3bd (0x17db bytes) 000468[1] 1775216891.510150 trunc from 0x7cec6d to 0x8f33fb (0x12478f bytes) 000470[1] 1775216891.592864 read 0x5f360c thru 0x5f4d53 (0x1748 bytes) 000471[1] 1775216891.613362 trunc from 0x8f33fc to 0x5cf1d5 (0x324226 bytes) 000476[1] 1775216891.697049 trunc from 0x5cf1d6 to 0x704b5d (0x135988 bytes) 000477[0] 1775216891.794851 trunc from 0x704b5e to 0x48f920 (0x27523d bytes) 000479[1] 1775216891.938453 write 0x54d275 thru 0x54eabb (0x1847 bytes) 000480[0] 1775216891.987095 write 0x12f007 thru 0x130345 (0x133f bytes) 000483[1] 1775216892.076261 write 0x05fdc4 thru 0x060833 (0xa70 bytes) 000484[1] 1775216892.164325 mapread 0x53cc8b thru 0x53ebc2 (0x1f38 bytes) 000486[1] 1775216892.196093 trunc from 0x54eabc to 0x70950d (0x1baa52 bytes) 000487[1] 1775216892.257919 mapread 0x329e5e thru 0x32bcae (0x1e51 bytes) 000491[1] 1775216892.348129 mapwrite 0x94598e thru 0x945a85 (0xf8 bytes) 000495[1] 1775216892.542569 trunc from 0x945a86 to 0x5b2ac6 (0x392fbf bytes) 000497[0] 1775216892.639204 mapwrite 0x15f734 thru 0x16162a (0x1ef7 bytes) 000499[0] 1775216892.818360 trunc from 0x5b2ac7 to 0x3f6041 (0x1bca85 bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 07:48:17 (1775216897) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 07:48:23 (1775216903) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.560593 s, 15.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.434483 s, 19.3 MB/s ldlm.namespaces.MGC192.168.204.131@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a3182cf2000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a318850d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a318850d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a318850d000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.2392 s, 6.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.4191 s, 23.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.60558 s, 40.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.9046 s, 15.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.61963 s, 10.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.30176 s, 24.4 MB/s PASS 16d (59s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 07:49:21 (1775216961) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.330668 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155413 s, 6.7 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 07:49:29 (1775216969) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 07:49:58 (1775216998) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 07:50:28 (1775217028) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 07:50:36 (1775217036) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 07:50:44 (1775217044) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16461 s, 8.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42735 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.42732 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.02782 s, 12.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.81174 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.3699 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.18448 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24774 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.41826 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.06219 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.20657 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.57328 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79838 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.3169 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49122 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18034 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60369 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.7848 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.19595 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 07:51:31 (1775217091) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Fri Apr 3 07:52:20 EDT 2026 + fsx 2 done: rc=0 Fri Apr 3 07:52:20 EDT 2026 + fsx 3 done: rc=0 Fri Apr 3 07:52:20 EDT 2026 + fsx 4 done: rc=0 Fri Apr 3 07:52:20 EDT 2026 + fsx 5 done: rc=0 Fri Apr 3 07:52:20 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 07:52:28 (1775217148) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 07:52:40 (1775217160) mmap test1: basic mmap operation (PASS, 0.1926s) mmap test2: MAP_PRIVATE not write back (PASS, 0.12311s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.6813s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6853s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.24749s) mmap test6: check mmap write/read content on two nodes (PASS, 0.2809s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.7832s) mmap test11: multi-thread shared mmap (PASS, 1.9964s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1234s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 07:53:08 (1775217188) oleg431-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 07:53:13 (1775217193) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 07:53:21 (1775217201) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 07:53:29 (1775217209) now is 1775217211 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 new atime is 1775217273 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 07:54:39 (1775217279) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 07:54:46 (1775217286) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 07:54:54 (1775217294) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 07:55:03 (1775217303) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 07:55:06 (1775217306) utime: good mknod times 1775217307 <= 1775217308 <= 1775217308 for /mnt/lustre/f26a utime: good mknod times 1775217307 <= 1775217308 <= 1775217308 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 07:55:15 (1775217315) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 07:55:25 (1775217325) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-03 07:55:27.000000000 -0400 Birth: 2026-04-03 07:55:26.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-03 07:55:27.000000000 -0400 Birth: 2026-04-03 07:55:26.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 07:55:35 (1775217335) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 07:55:45 (1775217345) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 07:55:53 (1775217353) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.131@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 07:56:14 (1775217374) -rw-r--r-- 1 root root 0 Apr 3 07:56 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 07:56:23 (1775217383) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42939 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 07:56:31 (1775217391) PASS 31t (8s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 07:56:40 (1775217400) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 07:56:43 (1775217403) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 07:56:47 (1775217407) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 07:56:51 (1775217411) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 07:56:54 (1775217414) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid in FULL state after 0 sec oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid in IDLE state after 0 sec oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid in IDLE state after 0 sec oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid in IDLE state after 0 sec oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a3182cf2000.ost_server_uuid in IDLE state after 0 sec oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a3182cf2000.ost_server_uuid in IDLE state after 0 sec PASS 34 (131s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 07:59:05 (1775217545) fail_loc=0x80000317 Race attempt 0 Wait for 48993 49025 for 60 sec... fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.131@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.131@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 569 (time 1775217561.76 total 10.00 last 56.88) - open/close 1083 (time 1775217571.77 total 20.02 last 51.31) - open/close 1599 (time 1775217581.79 total 30.04 last 51.49) - open/close 2160 (time 1775217591.80 total 40.05 last 56.03) - open/close 2733 (time 1775217601.81 total 50.06 last 57.27) - open/close 3291 (time 1775217611.82 total 60.07 last 55.75) - open/close 3806 (time 1775217621.83 total 70.08 last 51.46) total: 4000 open/close in 73.51 seconds: 54.42 ops/second PASS 35 (80s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:00:25 (1775217625) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.805885 s, 65.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7429120) after(7480320) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.873531 s, 60.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7480320) after_dd(7437312) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.786294 s, 66.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7430144) after(7481344) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.642275 s, 81.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7481344) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.814793 s, 64.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7487488) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.560507 s, 93.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.639662 s, 82.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7488512) after_dd(7432192) after(7483392) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.679777 s, 77.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7483392) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.560065 s, 93.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7426048) after(7477248) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.530202 s, 98.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7477248) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.919614 s, 57.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7440384) after(7491584) PASS 36 (308s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 08:05:33 (1775217933) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 - create 1115 (time 1775217947.43 total 10.01 last 111.40) - create 2396 (time 1775217957.43 total 20.01 last 128.03) - create 3536 (time 1775217967.43 total 30.02 last 113.98) - create 4591 (time 1775217977.44 total 40.02 last 105.44) - create 5564 (time 1775217987.44 total 50.02 last 97.29) - create 6403 (time 1775217997.45 total 60.03 last 83.83) - create 7272 (time 1775218007.45 total 70.03 last 86.89) - create 8197 (time 1775218017.45 total 80.03 last 92.49) - create 9264 (time 1775218027.46 total 90.04 last 106.63) total: 10000 create in 96.93 seconds: 103.17 ops/second PASS 37 (107s) == sanityn test 39a: file mtime does not change after rename ========================================================== 08:07:21 (1775218041) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 08:07:28 (1775218048) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 08:07:36 (1775218056) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 08:07:45 (1775218065) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 08:07:55 (1775218075) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 08:08:12 (1775218092) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 08:08:27 (1775218107) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 08:08:42 (1775218122) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 08:08:59 (1775218139) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 08:09:11 (1775218151) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 08:09:23 (1775218163) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 08:09:37 (1775218177) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 08:09:49 (1775218189) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 08:10:01 (1775218201) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 08:10:13 (1775218213) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 08:10:28 (1775218228) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 08:10:43 (1775218243) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1063s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 08:28:26 (1775219306) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 08:28:39 (1775219319) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 08:28:53 (1775219333) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 08:29:07 (1775219347) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 08:29:20 (1775219360) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 08:29:34 (1775219374) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 08:29:48 (1775219388) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 08:30:03 (1775219403) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 08:30:18 (1775219418) PASS 43a (78s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 08:31:36 (1775219496) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 08:31:51 (1775219511) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 08:32:05 (1775219525) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 08:32:19 (1775219539) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 08:32:32 (1775219552) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (11s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 08:32:43 (1775219563) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 08:32:55 (1775219575) Conflict PASS 43h (11s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 08:33:07 (1775219587) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 08:33:10 (1775219590) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (123s) == sanityn test 43k: unlink vs create ==================== 08:35:14 (1775219714) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1058s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 08:52:51 (1775220771) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Apr 3 08:52:57 EDT 2026 PASS 44a (9s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 08:53:00 (1775220780) open: File exists Conflict PASS 44b (9s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 08:53:09 (1775220789) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (9s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 08:53:18 (1775220798) Conflict PASS 44d (9s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 08:53:27 (1775220807) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 08:53:36 (1775220816) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 08:53:45 (1775220825) Conflict PASS 44g (9s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 08:53:54 (1775220834) Conflict PASS 44h (9s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 08:54:03 (1775220843) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 08:54:06 (1775220846) PASS 45a (81s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 08:55:26 (1775220926) Conflict PASS 45b (9s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 08:55:35 (1775220935) Conflict PASS 45c (8s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 08:55:44 (1775220944) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (9s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 08:55:52 (1775220952) Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 08:56:00 (1775220960) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (8s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 08:56:08 (1775220968) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 08:56:15 (1775220975) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 08:56:22 (1775220982) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 08:56:24 (1775220984) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (606s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 09:06:30 (1775221590) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 09:06:36 (1775221596) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 09:06:42 (1775221602) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 09:06:48 (1775221608) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 09:06:54 (1775221614) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 09:07:00 (1775221620) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 09:07:06 (1775221626) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 09:07:12 (1775221632) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 09:07:18 (1775221638) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 09:07:19 (1775221639) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 09:07:20 (1775221640) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 09:07:21 (1775221641) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 09:07:23 (1775221643) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 09:07:24 (1775221644) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 09:07:25 (1775221645) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 09:07:27 (1775221647) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 09:07:28 (1775221648) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0113403 s, 903 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 09:07:36 (1775221656) 0+1 records in 0+1 records out 158 bytes copied, 0.00484872 s, 32.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216619) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 09:07:41 (1775221661) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00505153 s, 203 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231687 s, 45.3 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 09:07:56 (1775221676) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00547182 s, 187 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 09:08:03 (1775221683) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118546 s, 88.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 09:08:06 (1775221686) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 09:08:11 (1775221691) llite.lustre-ffff8a3182cf2000.xattr_cache=1 llite.lustre-ffff8a318850d000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505008 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:11.000000000 -0400 Modify: 2026-04-03 09:08:11.000000000 -0400 Change: 2026-04-03 09:08:11.000000000 -0400 Birth: 2026-04-03 09:08:10.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:17.000000000 -0400 Modify: 2026-04-03 09:08:17.000000000 -0400 Change: 2026-04-03 09:08:17.000000000 -0400 Birth: 2026-04-03 09:08:15.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:22.000000000 -0400 Modify: 2026-04-03 09:08:22.000000000 -0400 Change: 2026-04-03 09:08:22.000000000 -0400 Birth: 2026-04-03 09:08:21.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:22.000000000 -0400 Modify: 2026-04-03 09:08:23.000000000 -0400 Change: 2026-04-03 09:08:23.000000000 -0400 Birth: 2026-04-03 09:08:21.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:28.000000000 -0400 Modify: 2026-04-03 09:08:28.000000000 -0400 Change: 2026-04-03 09:08:28.000000000 -0400 Birth: 2026-04-03 09:08:26.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:08:28.000000000 -0400 Modify: 2026-04-03 09:08:28.000000000 -0400 Change: 2026-04-03 09:08:28.000000000 -0400 Birth: 2026-04-03 09:08:26.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 09:08:35 (1775221715) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 09:08:43 (1775221723) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 09:08:51 (1775221731) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 09:09:04 (1775221744) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 09:09:14 (1775221754) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 09:09:15 (1775221755) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 09:09:28 (1775221768) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 09:09:42 (1775221782) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1775221783 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.54329 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (27s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 09:10:09 (1775221809) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1775221811 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.81482 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (37s) == sanityn test 60: Verify data_version behaviour ======== 09:10:46 (1775221846) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 09:10:49 (1775221849) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 09:10:52 (1775221852) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 09:10:55 (1775221855) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00351093 s, 18.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 09:10:57 (1775221857) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00366298 s, 17.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 09:10:58 (1775221858) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 09:10:59 (1775221859) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00540587 s, 12.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 09:11:00 (1775221860) llite.lustre-ffff8a3182cf2000.xattr_cache=1 llite.lustre-ffff8a318850d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 09:11:03 (1775221863) llite.lustre-ffff8a3182cf2000.xattr_cache=1 llite.lustre-ffff8a318850d000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 09:11:05 (1775221865) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00600699 s, 170 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00408412 s, 251 kB/s 229394: get lock2 229395: get lock1 229394: done 229394: get lock1 229395: done 229395: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229394: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 229395: cannot get lock2: Resource deadlock avoided 229395: exit rc=0 229394: done 229394: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 09:11:12 (1775221872) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0138296 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860751 s, 122 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229976 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 09:11:19 (1775221879) opening files: ........ closing files: ........ PASS 76 (16s) == sanityn test 77a: check FIFO NRS policy =============== 09:11:35 (1775221895) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0815291 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237584 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211937 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353569 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231497 s, 45.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249645 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272202 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503034 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058572 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388636 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068193 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482755 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688705 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716388 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884043 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963228 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663789 s, 15.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268667 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232927 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273224 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193062 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334437 s, 31.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269738 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258249 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022586 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159309 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278725 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153142 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162958 s, 64.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365481 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231874 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146162 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179673 s, 58.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 09:11:38 (1775221898) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0891153 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162948 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135891 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897207 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374281 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388286 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342324 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463256 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201547 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406203 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031623 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469738 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014954 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430036 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065957 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500342 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346957 s, 30.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229312 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222776 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260378 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207956 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233662 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153007 s, 68.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188448 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144669 s, 72.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280518 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307838 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281419 s, 37.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286365 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121858 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627617 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142834 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497621 s, 211 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0768637 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225912 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308799 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295118 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354734 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314753 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292079 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317465 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04689 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467828 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327279 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548472 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318268 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598859 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721084 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618606 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728424 s, 14.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243176 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368964 s, 284 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398918 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307261 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00268705 s, 390 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301892 s, 347 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330654 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03678 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408442 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248861 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241519 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414532 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193089 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175014 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145687 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143426 s, 73.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 09:11:43 (1775221903) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.101757 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182043 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020333 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335851 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212534 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400363 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039697 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041489 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400187 s, 26.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344099 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181531 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352639 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310634 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516283 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015365 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579941 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532583 s, 19.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145689 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193063 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179843 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434221 s, 241 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157943 s, 66.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011773 s, 89.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188562 s, 55.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254603 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106944 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255696 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720644 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194273 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259542 s, 40.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980639 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368111 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012679 s, 82.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.095109 s, 176 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174665 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231554 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196561 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237065 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03295 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447533 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347517 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430931 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472244 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214506 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405249 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361544 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423291 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330185 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465432 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407946 s, 25.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189601 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171693 s, 61.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173728 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219418 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225869 s, 46.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124058 s, 84.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151751 s, 69.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177707 s, 59.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281009 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347859 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120753 s, 86.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03589 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124562 s, 84.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607387 s, 173 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345548 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126524 s, 82.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0783646 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177359 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257939 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366384 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372115 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393297 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372487 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288255 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526084 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425085 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388174 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784853 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688394 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479443 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557939 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704295 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940442 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323739 s, 32.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280615 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276361 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259136 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204801 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254309 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183695 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226145 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128109 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347284 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199681 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325806 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367645 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354723 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439957 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027528 s, 38.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 09:11:48 (1775221908) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.103508 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289269 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435725 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385698 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578404 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482359 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445487 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472967 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614239 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548749 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389286 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668587 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047233 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368234 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628567 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747073 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844852 s, 12.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127806 s, 82.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230822 s, 45.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300788 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103764 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226763 s, 46.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205121 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298726 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173209 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326252 s, 32.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158436 s, 66.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290209 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277581 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026473 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309548 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532582 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015615 s, 67.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0894591 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218617 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223528 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237397 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464999 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302584 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247889 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409189 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317961 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228356 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909321 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535758 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909504 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944054 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898719 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822643 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111715 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304193 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273479 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334318 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356401 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223922 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224719 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325709 s, 32.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360941 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221697 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173834 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172637 s, 60.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402383 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366302 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206461 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215743 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202503 s, 51.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0855998 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132691 s, 79.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289223 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431926 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031423 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261564 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048688 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539877 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560248 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588572 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655083 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663776 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450707 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582651 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552077 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370345 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647764 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225828 s, 46.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299329 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237676 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311196 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406734 s, 25.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031887 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290794 s, 36.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374489 s, 28.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291457 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349813 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357816 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109737 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133637 s, 78.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325818 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196847 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314025 s, 33.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 09:11:54 (1775221914) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.31@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0744137 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147019 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161556 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460298 s, 22.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049706 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502487 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483287 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347884 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397599 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335872 s, 31.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342583 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401626 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492305 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608092 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382623 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611835 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582592 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167873 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188181 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105805 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162853 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147258 s, 71.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188413 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143056 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021181 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287227 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361675 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231424 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00730491 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213966 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241714 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235326 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00274383 s, 382 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.069175 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187281 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300399 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276734 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445417 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358931 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324002 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042386 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469456 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313738 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422423 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413509 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496899 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490968 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491425 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636663 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639249 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226308 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164239 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235343 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231953 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260902 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180264 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239339 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270677 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194486 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132785 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403016 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267971 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298559 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628647 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132268 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647944 s, 162 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0959575 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194391 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394315 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323342 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323507 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402816 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516315 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557828 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489736 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298406 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503334 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529252 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651499 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716979 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388445 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776438 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560698 s, 18.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189492 s, 55.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321151 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271727 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326226 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214158 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350251 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220561 s, 47.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255215 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286594 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255775 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407801 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308952 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117935 s, 88.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497455 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038873 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694534 s, 151 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.079272 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173913 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347934 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370167 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328832 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369293 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535032 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328851 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384214 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446571 s, 23.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059804 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546708 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659441 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776669 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774796 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678432 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814118 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144764 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176979 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159203 s, 65.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139549 s, 75.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254438 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731553 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234616 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242157 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301672 s, 348 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134031 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021421 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264541 s, 39.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262985 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299432 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00564227 s, 186 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145526 s, 72.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 09:12:03 (1775221923) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.208428 s, 80.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150023 s, 69.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187806 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275005 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038525 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509336 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477417 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565246 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530627 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510696 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551304 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653994 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062228 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525115 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616029 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595453 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646233 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011022 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200102 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203169 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237255 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193213 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197712 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175153 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177267 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265808 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223613 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363061 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235951 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234673 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02279 s, 46.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025221 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259052 s, 40.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0737075 s, 228 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989956 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014679 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262731 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203735 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441495 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029853 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388025 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295201 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302212 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555004 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494005 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350228 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431486 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294281 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363402 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586415 s, 17.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144573 s, 72.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481594 s, 218 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183528 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106724 s, 98.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236146 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103172 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227665 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407366 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003965 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397215 s, 264 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445117 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418035 s, 251 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721174 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372953 s, 281 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367907 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707295 s, 148 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0912224 s, 184 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117928 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377157 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394131 s, 26.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374975 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483477 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490864 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282793 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035623 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066396 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461501 s, 22.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728261 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599985 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522025 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634296 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564812 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529963 s, 19.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179455 s, 58.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205247 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258427 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191178 s, 54.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243139 s, 43.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266863 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254711 s, 41.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237172 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021545 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253527 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103342 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026501 s, 39.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245538 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226064 s, 46.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307847 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00555163 s, 189 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0760719 s, 221 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994133 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272953 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032277 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177554 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246669 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257356 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312504 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496499 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136438 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512974 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510208 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451443 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521073 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611517 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445577 s, 23.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a3182cf2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607752 s, 17.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145798 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171651 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198716 s, 52.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8a3182cf2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256315 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00775836 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157101 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265018 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182382 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298785 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219412 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183434 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00701018 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346149 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251519 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328423 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00530536 s, 198 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 09:12:11 (1775221931) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 09:12:15 (1775221935) oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg431-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg431-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg431-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg431-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 09:12:19 (1775221939) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 09:12:26 (1775221946) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0635522 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184849 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232225 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367946 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304148 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439205 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409558 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388749 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277149 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348091 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399071 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302089 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304776 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432769 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377189 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352152 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631546 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125164 s, 83.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149955 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121784 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182008 s, 57.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014297 s, 73.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086329 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00652033 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161299 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276648 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220016 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225078 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262052 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795763 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351582 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377646 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262338 s, 40.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96053 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.44 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (53s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 09:13:19 (1775221999) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21678 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242494 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320324 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148074 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029671 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223212 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226428 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260943 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219338 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162678 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159811 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150665 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160514 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156828 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182012 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183858 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196565 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230689 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232544 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274535 s, 38.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169337 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202434 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313794 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204754 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298905 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027754 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295094 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348234 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278035 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303778 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308848 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365268 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340032 s, 30.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8362 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9662 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28115 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476802 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336359 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686864 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777715 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046494 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650431 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138182 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142422 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150235 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137228 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162702 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184841 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187424 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201377 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213938 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185362 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114866 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140066 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13049 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154183 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15692 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951431 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143491 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155686 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169148 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141246 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146232 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166011 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157084 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199061 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197306 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173484 s, 6.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.235 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7773 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (141s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 09:15:40 (1775222140) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23525 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170812 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188431 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536662 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613866 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464121 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618204 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781599 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744758 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156487 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158503 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177966 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163484 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181485 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179839 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173011 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206043 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285211 s, 36.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240464 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390608 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531581 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046349 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346375 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335871 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387467 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620289 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408567 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607465 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048151 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610321 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414022 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425698 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053516 s, 19.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8564 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9305 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23825 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309982 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294479 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625951 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183371 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398821 s, 26.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631689 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766011 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658981 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144617 s, 7.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149249 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15836 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175669 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176114 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193496 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18892 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210613 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282206 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030035 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330808 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318671 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218666 s, 48.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249201 s, 42.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354825 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397538 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310168 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352821 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293822 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288968 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334955 s, 31.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287954 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320937 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340485 s, 30.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8336 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.954 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (137s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 09:17:57 (1775222277) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.304287 s, 55.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159878 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133601 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479511 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498144 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471119 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534622 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721916 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744319 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078563 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638189 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713849 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798407 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812116 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715498 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084568 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840196 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222368 s, 47.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207865 s, 50.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153055 s, 68.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237686 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217643 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230058 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125784 s, 83.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204845 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212035 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274901 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159113 s, 65.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178601 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221222 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215866 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02503 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050269 s, 209 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93452 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74889 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.31@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.31@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354792 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408985 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560293 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569544 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153896 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184817 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165599 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177237 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260764 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272556 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211314 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256697 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18581 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279245 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255052 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332962 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288474 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617524 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549433 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547774 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592997 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096481 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455683 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053561 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538687 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489508 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792821 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716408 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727498 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728994 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065088 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737176 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779597 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5262 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94928 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.31@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.418289 s, 40.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217476 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624429 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665794 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131017 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11746 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141676 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177352 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147111 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17121 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175321 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153162 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157076 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125309 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17658 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144913 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205282 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132124 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481218 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217316 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432438 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520623 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00604928 s, 173 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434988 s, 241 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479474 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827982 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616369 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453415 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521782 s, 201 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499992 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697482 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303745 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441354 s, 23.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.14711 s, 17.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97634 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199313 s, 84.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159689 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208071 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342498 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642291 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795278 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764249 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701394 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898115 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760452 s, 13.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838251 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887199 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675472 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153913 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176237 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203485 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216967 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047252 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519696 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486568 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00553987 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642714 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537001 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478124 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107449 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427281 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319998 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339388 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431796 s, 24.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313058 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318364 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235787 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283707 s, 37.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91636 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82662 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51017 s, 19.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82656 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193046 s, 86.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117862 s, 89.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018618 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477615 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503945 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377518 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523721 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314729 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132473 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721499 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582646 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079218 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884286 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796621 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855961 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106339 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103983 s, 10.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00261711 s, 401 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388987 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455085 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337364 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510704 s, 205 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546783 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161804 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334926 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395085 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305912 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638333 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215506 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017354 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107733 s, 97.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630996 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108256 s, 96.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6754 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5267 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320303 s, 52.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264042 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298508 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236302 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371855 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449152 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317853 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864962 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136524 s, 76.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173078 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118037 s, 88.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343626 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654727 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791302 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113082 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116754 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093324 s, 11.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363742 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861187 s, 122 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018329 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262857 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356503 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680122 s, 15.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0031938 s, 328 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868755 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473393 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550067 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519968 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460841 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615767 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371413 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341171 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027501 s, 38.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.32442 s, 16.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84307 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.209646 s, 80.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399427 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042041 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592114 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162171 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063223 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812021 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883344 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074132 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558414 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439405 s, 23.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924849 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612752 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693767 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878837 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139365 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173457 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193144 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305053 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033968 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053671 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437333 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521047 s, 201 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605414 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414033 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577274 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313653 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085859 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579651 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783426 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491927 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115611 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156623 s, 66.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.089 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88379 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9654 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86906 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (388s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 09:24:25 (1775222665) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.284199 s, 59.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169121 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184049 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297588 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373289 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899781 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113082 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08948 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121324 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101132 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951962 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112971 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129159 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108107 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970772 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105196 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143659 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332554 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029153 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391858 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364696 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426669 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419969 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400425 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335655 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375443 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03818 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336614 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401257 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394024 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452312 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507541 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491161 s, 21.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35191 s, 19.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82814 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (40s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 09:25:06 (1775222706) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.31@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.31@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.323466 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223228 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267223 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759293 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424507 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745315 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454385 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601638 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715397 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629408 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726362 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622956 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935145 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241205 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220008 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266404 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270051 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200798 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258341 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424729 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257483 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257843 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032235 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330281 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343016 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350069 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352923 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363118 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370728 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389458 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037314 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583958 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420418 s, 24.9 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31223 s, 19.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81448 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (42s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 09:25:47 (1775222747) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.31@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.290565 s, 57.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384691 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052138 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637844 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575508 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674036 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640282 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677391 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13472 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718107 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101726 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129878 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943241 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13606 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143987 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127042 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387219 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380498 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379925 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473826 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381896 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396717 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422186 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677968 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473486 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432223 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492978 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294196 s, 35.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436788 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522439 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547472 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573523 s, 18.3 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.92129 s, 17.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98351 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 09:26:23 (1775222783) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.288065 s, 58.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428653 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563046 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463374 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416219 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752796 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082682 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789218 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787395 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424574 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552959 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119558 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853954 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895737 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134247 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268322 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303949 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546723 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464055 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292134 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977559 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121317 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491463 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496457 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618003 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044138 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534348 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122768 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552126 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407944 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560109 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559908 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602533 s, 174 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99597 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81615 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34828 s, 19.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8147 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (80s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 09:27:43 (1775222863) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.923597 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234057 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347509 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359217 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469253 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495271 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493573 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803696 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783912 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132693 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176654 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178626 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16331 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190079 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184802 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200399 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216544 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480849 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019212 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436085 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217958 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314252 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800804 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550343 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334425 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331408 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055907 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372214 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599577 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376912 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388601 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397332 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401534 s, 26.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6599 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8965 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 09:28:50 (1775222930) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276269 s, 60.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331515 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336982 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794082 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08607 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891412 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867462 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798954 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917552 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103953 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944521 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742134 s, 14.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909946 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940105 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072537 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100227 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126474 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114575 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395125 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045087 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515298 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574248 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003634 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402066 s, 261 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412837 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462461 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355038 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560791 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279826 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243757 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839283 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929054 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716904 s, 146 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29216 s, 19.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83527 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg431-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146686 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176655 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957677 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296757 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459173 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491538 s, 21.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379252 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600973 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471576 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047694 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604921 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487294 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668238 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546767 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209229 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212478 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215431 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490592 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252839 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505096 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544623 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489885 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376266 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488517 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604303 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645475 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581998 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637381 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621823 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732559 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642755 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763321 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752151 s, 13.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92504 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83549 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93304 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84432 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (122s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 09:30:52 (1775223052) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre opts: Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg431-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg431-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.31,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg431-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.31,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg431-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.31,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/62056/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.31,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg431-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Started clients oleg431-client.virtnet: 192.168.204.131@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Started clients oleg431-client.virtnet: 192.168.204.131@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a3183fac800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a31c0d79800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a3183fac800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a31c0d79800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08127 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518761 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464355 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431887 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191369 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497866 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720847 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737423 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117926 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13312 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414314 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424552 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348502 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390692 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428359 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349789 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454041 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389289 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447992 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376929 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449073 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.55457 s, 675 kB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467682 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390907 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.56335 s, 671 kB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43019 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419416 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.61792 s, 648 kB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.63429 s, 642 kB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470703 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.65698 s, 633 kB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.65657 s, 633 kB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.70208 s, 616 kB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5671 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.72174 s, 18.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.47164 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236545 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246914 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421771 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315056 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398656 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517771 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418785 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515863 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532342 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.687278 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.640363 s, 1.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.705809 s, 1.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.696557 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.693981 s, 1.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.71205 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.71658 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233587 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16703 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208148 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198349 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251636 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409874 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223903 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244106 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251647 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262844 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270132 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307838 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313533 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36515 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368967 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403512 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a31c0d79800.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1616 s, 8.0 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.017 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (201s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 09:34:13 (1775223253) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg431-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 09:34:30 (1775223270) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 09:34:40 (1775223280) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg431-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.66784 s, 286 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg431-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.139 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 09:35:38 (1775223338) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.307773 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816277 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895563 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197784 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199689 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242492 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241792 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263685 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287756 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27329 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262118 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267587 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256631 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281559 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330101 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293279 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301029 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a31c0d79800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8a31c0d79800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934856 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587644 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778029 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115378 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107462 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821878 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655116 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737999 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870627 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840056 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843878 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794942 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857272 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101411 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109728 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932141 s, 11.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg431-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg431-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.43086 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29823 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.548906 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708176 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132944 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121989 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130008 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122343 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120654 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184583 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143737 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225818 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205716 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2224 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1793 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171913 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178224 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190202 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225853 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078262 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904225 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100088 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106559 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863164 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788309 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971214 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112207 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103602 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919265 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102643 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160381 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910775 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131216 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117295 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152703 s, 6.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg431-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg431-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97367 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1888 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 09:37:09 (1775223429) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 09:37:23 (1775223443) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (126s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 09:39:28 (1775223568) mds.MDS.mdt.nrs_policies=tbf oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg431-client: oleg431-server: ssh exited with exit code 36 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 oleg431-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg431-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 09:40:06 (1775223606) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 268322 kill md_thread xargs PASS 77r (56s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 09:41:02 (1775223662) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 09:41:13 (1775223673) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826934665 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:41:15.000000000 -0400 Modify: 2026-04-03 09:41:15.000000000 -0400 Change: 2026-04-03 09:41:15.000000000 -0400 Birth: 2026-04-03 09:41:14.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 09:41:32 (1775223692) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 09:41:36 (1775223696) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 09:41:39 (1775223699) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 09:41:44 (1775223704) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 09:41:47 (1775223707) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 09:41:51 (1775223711) - open/close 662 (time 1775223726.16 total 10.01 last 66.13) - open/close 1285 (time 1775223736.17 total 20.02 last 62.24) - open/close 1884 (time 1775223746.19 total 30.04 last 59.78) - open/close 2453 (time 1775223756.19 total 40.04 last 56.89) total: 2500 open/close in 40.78 seconds: 61.30 ops/second snapshot_time 1775223852.748712845 secs.nsecs start_time 1775223712.630697981 secs.nsecs elapsed_time 140.118014864 secs.nsecs open 2505 samples [usecs] 167 21839 5011054 16169262156 close 2504 samples [usecs] 224 7083 1255528 1337128566 mknod 2501 samples [usecs] 666 21719 4401700 12711509936 unlink 2500 samples [usecs] 817 57340 6553258 51282227158 mkdir 2 samples [usecs] 1640 2642 4282 9669764 rmdir 2 samples [usecs] 18467 20844 39311 775502425 rename 2500 samples [usecs] 548 49799 18777575 262231656915 getattr 3762 samples [usecs] 7 5355 268932 238604574 parallel_rename_file 2500 samples [usecs] 548 49799 18777575 262231656915 crossdir_rename 2500 samples [usecs] 548 49799 18777575 262231656915 PASS 81d (149s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 09:44:19 (1775223859) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7563 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7563 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 09:44:25 (1775223865) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 09:44:29 (1775223869) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 09:44:42 (1775223882) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1775223883 Finishing test test1 at 1775223888 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 09:44:53 (1775223893) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 09:44:56 (1775223896) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 09:45:00 (1775223900) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 09:45:03 (1775223903) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5306 0x14ba 0x240000400 1 5269 0x1495 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5270 0x1496 0x280000400 0 5307 0x14bb 0x240000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 09:45:21 (1775223921) osc.lustre-OST0000-osc-ffff8a3183fac800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8a31c0d79800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a3183fac800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a31c0d79800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.24773 s, 33.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00693 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 283182 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0887315 s, 46.2 kB/s osc.lustre-OST0000-osc-ffff8a3183fac800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a31c0d79800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a3183fac800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a31c0d79800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 09:45:37 (1775223937) llite.lustre-ffff8a3183fac800.fast_read=0 llite.lustre-ffff8a31c0d79800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0661087 s, 31.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8a3183fac800.fast_read=1 llite.lustre-ffff8a31c0d79800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 09:45:58 (1775223958) llite.lustre-ffff8a3183fac800.fast_read=0 llite.lustre-ffff8a31c0d79800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00883387 s, 1.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8a3183fac800.fast_read=1 llite.lustre-ffff8a31c0d79800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 09:46:11 (1775223971) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 09:46:14 (1775223974) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0163206 s, 251 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 09:46:22 (1775223982) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208532 s, 10.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 09:46:30 (1775223990) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191187 s, 11.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 09:46:38 (1775223998) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0138179 s, 868 kB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 09:46:45 (1775224005) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00717737 s, 571 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 09:46:52 (1775224012) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0154995 s, 264 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 09:46:59 (1775224019) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00678179 s, 604 kB/s PASS 101c (8s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 09:47:09 (1775224029) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1775224032 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1775224036 Finishing test test23 at 1775224036 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 09:47:23 (1775224043) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 09:47:27 (1775224047) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 09:47:55 (1775224075) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 09:47:58 (1775224078) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.021776 s, 47.0 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937200 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:48:00.000000000 -0400 Modify: 2026-04-03 09:48:00.000000000 -0400 Change: 2026-04-03 09:48:00.000000000 -0400 Birth: 2026-04-03 09:47:58.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826937200 -rw-r--r-- 1 0 0 1775224078 1775224080 1775224080 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937200 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 09:48:00.000000000 -0400 Modify: 2026-04-03 09:48:00.000000000 -0400 Change: 2026-04-03 09:48:00.000000000 -0400 Birth: 2026-04-03 09:47:58.000000000 -0400 1775224080 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 09:48:06 (1775224086) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 09:48:13 (1775224093) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351358 s, 29.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5315 0x14c3 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 09:48:22 (1775224102) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.357831 s, 29.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5277 0x149d 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 09:48:37 (1775224117) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00522954 s, 1.9 MB/s llite.lustre-ffff8a3183fac800.hybrid_io=0 llite.lustre-ffff8a31c0d79800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.75774 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 09:48:48 (1775224128) 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.131@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 PASS 109 (1065s) == sanityn test 110: do not grant another lock on resend ========================================================== 10:06:33 (1775225193) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 10:06:35.000000000 -0400 Modify: 2026-04-03 10:06:35.000000000 -0400 Change: 2026-04-03 10:06:35.000000000 -0400 Birth: 2026-04-03 10:06:33.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg431-client.virtnet oleg431-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 10:07:56 (1775225276) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 10:07:59 (1775225279) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 10:08:01 (1775225281) lustre-OST0000-osc-ffff8a31a089a800 active. lustre-OST0001-osc-ffff8a31a089a800 active. lustre-OST0000-osc-ffff8a31a089a800 active. lustre-OST0001-osc-ffff8a31a089a800 active. MGC192.168.204.131@tcp active. PASS 113 (5s) == sanityn test 114: implicit default LMV inherit ======== 10:08:06 (1775225286) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 10:08:08 (1775225288) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 10:08:10 (1775225290) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 10:08:13 (1775225293) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0157757 s, 1.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0702247 s, 292 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (7s) == sanityn test 200: service remains healthy while able to process request ========================================================== 10:08:20 (1775225300) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0547 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1275 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1498 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1561 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1418 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1646 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.19 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0721 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.149 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1609 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1907 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1747 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1638 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1764 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2563 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3191 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.254 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2759 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a31a089a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a31a089a800.ost_server_uuid in FULL state after 0 sec PASS 200 (71s) cleanup: ====================================================== == sanityn test complete, duration 9055 sec ============== 10:09:32 (1775225372) === sanityn: start cleanup 10:09:33 (1775225373) === Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg431-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 10:12:13 (1775225533) ===