-----============= acceptance-small: sanityn ============----- Tue Sep 29 22:12:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg121-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 22:12:27 (1790734347) === Starting client oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Mount client oleg121-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Started clients oleg121-client.virtnet: 192.168.201.121@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg121-client.virtnet: executing check_config_client /mnt/lustre oleg121-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg121-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8acb8573d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8acb910e1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb8573d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb910e1000.idle_timeout=debug disable quota as required oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 22:13:14 (1790734394) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396039 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 22:13:17 (1790734397) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 22:13:27 (1790734407) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 22:13:35 (1790734415) -rw-r--r-- 1 root root 0 Sep 29 22:13 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 22:13:43 (1790734423) -rw-r--r-- 1 root root 0 Sep 29 22:13 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 22:13:51 (1790734431) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 22:13:59 (1790734439) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 22:14:06 (1790734446) total 1 -rwxrwxrwx 1 root root 0 Sep 29 22:13 f2a -rwxrwxrwx 1 root root 0 Sep 29 22:13 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 29 22:13 f2a -rwxrwxrwx 1 root root 0 Sep 29 22:13 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00935967 s, 54.7 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 22:14:14 (1790734454) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 22:14:18 (1790734458) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.614542 s, 3.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 22:14:26 (1790734466) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 22:14:34 (1790734474) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 22:14:43 (1790734483) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 22:14:51 (1790734491) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 22:14:59 (1790734499) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 22:15:07 (1790734507) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 22:15:14 (1790734514) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 22:15:22 (1790734522) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689652 s, 15.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0382281 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 22:15:29 (1790734529) 1+0 records in 1+0 records out 1 byte copied, 0.0160378 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113753 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123156 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0164024 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116997 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116952 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0134489 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119338 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0227666 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121856 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0202808 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0230356 s, 0.0 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 22:15:38 (1790734538) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0176732 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0506193 s, 80.9 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000241695 s, 12.7 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 22:15:45 (1790734545) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (10s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 22:15:55 (1790734555) warning: '-runtime' deprecated, use '-t runtime' instead - link 294 (time 1790734568.92 total 10.05 last 29.26) - link 564 (time 1790734578.95 total 20.08 last 26.90) - link 832 (time 1790734588.99 total 30.12 last 26.70) - link 1117 (time 1790734599.02 total 40.15 last 28.42) - link 1442 (time 1790734609.04 total 50.17 last 32.44) - link 1742 (time 1790734619.05 total 60.18 last 29.98) - link 2044 (time 1790734629.05 total 70.19 last 30.17) - link 2323 (time 1790734639.07 total 80.20 last 27.86) - link 2615 (time 1790734649.09 total 90.23 last 29.12) total: 2908 link in 100.00 seconds: 29.08 ops/second using seed 2796171503 running for 100 seconds - stat 4621 (time 1790734570 ; total 11 ; last 11) - stat 9615 (time 1790734581 ; total 22 ; last 11) - stat 10000 (time 1790734581 ; total 22 ; last 0) - stat 14636 (time 1790734592 ; total 33 ; last 11) - stat 20000 (time 1790734602 ; total 43 ; last 10) - stat 25147 (time 1790734613 ; total 54 ; last 11) - stat 28195 (time 1790734624 ; total 65 ; last 11) - stat 30000 (time 1790734630 ; total 71 ; last 6) - stat 32552 (time 1790734641 ; total 82 ; last 11) - stat 35356 (time 1790734652 ; total 93 ; last 11) total: 37281 stats in 100 seconds: 372.809998 stats/second - unlinked 0 (time 1790734609 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2908) error: No such file or directory total: 2907 unlinks in 84 seconds: 34.607143 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21443) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21446) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 22:18:19 (1790734699) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 22:18:29 (1790734709) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 22631 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 22:18:37 (1790734717) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 23211 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 22:18:45 (1790734725) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 23789 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 22:18:53 (1790734733) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 24371 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 22:19:01 (1790734741) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 24953 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 22:19:10 (1790734750) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (60s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 22:20:11 (1790734811) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.46727 s, 18.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.416776 s, 20.1 MB/s Chance of close/open is 1 in 50 Seed set to 4816 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1790734817.004842 write 0x7520bb thru 0x75480a (0x2750 bytes) 000003[1] 1790734817.035280 read 0x41fcfe thru 0x42d663 (0xd966 bytes) truncating to largest ever: 0x21b375 000005[0] 1790734817.099078 trunc from 0x75480b to 0x21b374 (0x539496 bytes) 000007[0] 1790734817.152984 mapwrite 0x44a56f thru 0x44e48c (0x3f1e bytes) 000008[0] 1790734817.372993 punch from 0x307814 to 0x30ef1c (0x7709 bytes) 000011[1] 1790734817.451397 mapwrite 0x4cf336 thru 0x4d3284 (0x3f4f bytes) 000012[1] 1790734817.598088 punch from 0x065dfb to 0x06dd57 (0x7f5d bytes) truncating to largest ever: 0x38365f 000014[0] 1790734817.605952 trunc from 0x4d3285 to 0x38365e (0x14fc26 bytes) 000015[1] 1790734817.702042 trunc from 0x38365f to 0x0c84fc (0x2bb162 bytes) 000016[0] 1790734817.807108 punch from 0x085e3c to 0x0933fc (0xd5c1 bytes) 000017[0] 1790734817.892783 punch from 0x0c4684 to 0x0c84fc (0x3e79 bytes) 000018[1] 1790734817.916739 mapwrite 0x6f8195 thru 0x6fd53d (0x53a9 bytes) 000020[0] 1790734818.100037 write 0x28ae55 thru 0x28fa06 (0x4bb2 bytes) truncating to largest ever: 0x51a913 000023[1] 1790734818.196153 trunc from 0x6fd53e to 0x51a912 (0x1e2c2b bytes) 000026[0] 1790734818.309018 trunc from 0x51a913 to 0x1b8b8a (0x361d88 bytes) 000027[0] 1790734818.387627 mapread 0x11da51 thru 0x124ab8 (0x7068 bytes) 000028[0] 1790734818.417709 mapread 0x13a598 thru 0x13afc1 (0xa2a bytes) 000031[0] 1790734818.480850 trunc from 0x1b8b8b to 0x480a6c (0x2c7ee2 bytes) 000033[0] 1790734818.579007 write 0x38ec0a thru 0x39bf84 (0xd37b bytes) 000035[1] 1790734818.612020 write 0x1eff6b thru 0x1f60c7 (0x615d bytes) 000037[0] 1790734818.718101 write 0x9ae6aa thru 0x9ba5c1 (0xbf18 bytes) 000038[0] 1790734818.790925 read 0x33d270 thru 0x33ddb3 (0xb44 bytes) 000039[0] 1790734818.797815 write 0x748fc2 thru 0x74a872 (0x18b1 bytes) 000046[1] 1790734818.937559 mapread 0x8786be thru 0x886f10 (0xe853 bytes) 000047[1] 1790734819.091364 trunc from 0x9ba5c2 to 0x08ba16 (0x92ebab bytes) 000049[1] 1790734819.186378 punch from 0x023bf6 to 0x02b645 (0x7a50 bytes) 000050[1] 1790734819.197625 mapread 0x06d6a9 thru 0x070217 (0x2b6f bytes) 000053[0] 1790734819.282554 read 0x0876b4 thru 0x08ba16 (0x4363 bytes) 000054[1] 1790734819.423848 mapwrite 0x3518b7 thru 0x35edc1 (0xd50b bytes) 000056[0] 1790734819.679606 read 0x33e396 thru 0x34a669 (0xc2d4 bytes) 000057[0] 1790734819.803284 punch from 0x18292d to 0x1911fb (0xe8cf bytes) 000058[1] 1790734819.859986 read 0x0cf709 thru 0x0d1303 (0x1bfb bytes) 000059[1] 1790734819.908919 read 0x2ac964 thru 0x2bbe16 (0xf4b3 bytes) 000060[1] 1790734820.018266 write 0x037f19 thru 0x046bd1 (0xecb9 bytes) 000061[1] 1790734820.084235 trunc from 0x35edc2 to 0x06c43f (0x2f2982 bytes) 000063[0] 1790734820.177909 punch from 0x025838 to 0x0318e7 (0xc0b0 bytes) 000064[1] 1790734820.231326 read 0x060e81 thru 0x066565 (0x56e5 bytes) 000066[0] 1790734820.300388 mapwrite 0x41ae4b thru 0x4205ca (0x5780 bytes) truncating to largest ever: 0x836655 000067[0] 1790734820.629403 trunc from 0x4205cb to 0x836654 (0x41608a bytes) 000068[1] 1790734820.682684 read 0x14bf82 thru 0x153ed6 (0x7f55 bytes) 000070[0] 1790734820.780880 mapread 0x72580a thru 0x7293c7 (0x3bbe bytes) 000071[1] 1790734820.894732 mapread 0x5837d3 thru 0x58af24 (0x7752 bytes) 000072[0] 1790734820.936300 punch from 0x27a7a9 to 0x27e667 (0x3ebf bytes) 000073[0] 1790734820.967297 mapread 0x31f214 thru 0x32af0b (0xbcf8 bytes) 000074[0] 1790734821.084880 read 0x2a91a2 thru 0x2aaebd (0x1d1c bytes) 000075[0] 1790734821.091016 mapread 0x123f37 thru 0x132cad (0xed77 bytes) 000078[0] 1790734821.167548 trunc from 0x836655 to 0x1d2e88 (0x6637cc bytes) 000081[0] 1790734821.335748 mapwrite 0x9c61c5 thru 0x9d2731 (0xc56d bytes) 000083[1] 1790734821.477954 write 0x5ed20a thru 0x5fd199 (0xff90 bytes) 000084[0] 1790734821.526876 mapread 0x68d1fb thru 0x698a5d (0xb863 bytes) 000085[0] 1790734821.591266 mapread 0x0f0f61 thru 0x0f4bb7 (0x3c57 bytes) 000086[1] 1790734821.607404 mapread 0x786c6f thru 0x78da95 (0x6e27 bytes) 000087[0] 1790734821.648801 write 0x4c17e9 thru 0x4cfb63 (0xe37b bytes) truncating to largest ever: 0x9b8504 000089[1] 1790734821.698450 trunc from 0x9d2732 to 0x9b8503 (0x1a22e bytes) 000090[0] 1790734821.757869 punch from 0x61f94d to 0x62446e (0x4b22 bytes) 000091[0] 1790734821.794785 punch from 0x27ce0d to 0x28c6e9 (0xf8dd bytes) 000093[1] 1790734821.835525 mapread 0x39fceb thru 0x3a0657 (0x96d bytes) 000094[1] 1790734821.899774 punch from 0x5664f1 to 0x5700ed (0x9bfd bytes) 000095[1] 1790734821.926570 write 0x6b6c82 thru 0x6bc97a (0x5cf9 bytes) 000097[1] 1790734821.965275 read 0x4a1f2d thru 0x4b1091 (0xf165 bytes) 000098[1] 1790734822.001029 read 0x055e5b thru 0x059bcd (0x3d73 bytes) 000099[1] 1790734822.029728 mapread 0x2b2a34 thru 0x2bb097 (0x8664 bytes) 000100[1] 1790734822.063993 mapwrite 0x0f1588 thru 0x0ff9db (0xe454 bytes) 000101[1] 1790734822.184015 mapwrite 0x42b1d0 thru 0x42d521 (0x2352 bytes) 000102[0] 1790734822.296490 mapread 0x7f0cd9 thru 0x7f321a (0x2542 bytes) 000105[0] 1790734822.453711 read 0x6d465c thru 0x6d8e32 (0x47d7 bytes) 000106[1] 1790734822.467264 trunc from 0x9b8504 to 0x797acd (0x220a36 bytes) 000109[0] 1790734822.549576 trunc from 0x797ace to 0x1ee459 (0x5a9674 bytes) 000110[1] 1790734822.606956 punch from 0x040ad7 to 0x042e42 (0x236c bytes) 000112[1] 1790734822.662178 mapwrite 0x796251 thru 0x79ea05 (0x87b5 bytes) 000113[0] 1790734822.885256 mapread 0x14edb1 thru 0x151274 (0x24c4 bytes) 000114[1] 1790734823.007723 write 0x4e02a6 thru 0x4e5dcb (0x5b26 bytes) 000122[1] 1790734823.209084 trunc from 0x79ea06 to 0x95b877 (0x1bce72 bytes) 000123[0] 1790734823.266741 mapwrite 0x8c9daa thru 0x8ccd22 (0x2f79 bytes) 000124[0] 1790734823.414108 mapwrite 0x2220e3 thru 0x227fcf (0x5eed bytes) 000125[0] 1790734823.609759 trunc from 0x95b878 to 0x197814 (0x7c4063 bytes) 000126[0] 1790734823.765463 mapwrite 0x3a902a thru 0x3b5d88 (0xcd5f bytes) 000128[0] 1790734824.022826 punch from 0x199c54 to 0x1a2fcd (0x937a bytes) 000129[0] 1790734824.072837 write 0x05d321 thru 0x05dc7d (0x95d bytes) 000130[1] 1790734824.130520 write 0x39e593 thru 0x3a9a43 (0xb4b1 bytes) 000133[0] 1790734824.244597 write 0x99267b thru 0x99d15e (0xaae4 bytes) 000135[0] 1790734824.273199 punch from 0x055ae1 to 0x057d81 (0x22a1 bytes) 000136[1] 1790734824.282380 mapwrite 0x45329b thru 0x45cc4e (0x99b4 bytes) 000137[0] 1790734824.720550 read 0x213f42 thru 0x217a3d (0x3afc bytes) 000139[1] 1790734824.813789 trunc from 0x99d15f to 0x778fea (0x224174 bytes) 000140[0] 1790734824.932430 write 0x3b4780 thru 0x3b749f (0x2d20 bytes) 000143[0] 1790734825.217744 punch from 0x0a1db9 to 0x0a3d7a (0x1fc2 bytes) 000144[1] 1790734825.292840 mapwrite 0x785750 thru 0x790b60 (0xb411 bytes) 000146[1] 1790734825.631689 trunc from 0x790b61 to 0x6c8cde (0xc7e82 bytes) 000147[0] 1790734825.686235 mapread 0x57a9ca thru 0x582f78 (0x85af bytes) 000149[0] 1790734825.811584 mapwrite 0x19e4c6 thru 0x19f747 (0x1282 bytes) 000151[0] 1790734825.979923 mapread 0x5c329e thru 0x5c7897 (0x45fa bytes) 000152[0] 1790734826.000781 write 0x7d9536 thru 0x7de748 (0x5213 bytes) 000154[0] 1790734826.071668 punch from 0x195a6e to 0x1983fe (0x2991 bytes) 000157[0] 1790734826.121746 read 0x1192c1 thru 0x11b5ed (0x232d bytes) 000162[0] 1790734826.144035 mapread 0x589085 thru 0x591968 (0x88e4 bytes) 000163[1] 1790734826.258803 punch from 0x100b0c to 0x109938 (0x8e2d bytes) 000165[0] 1790734826.366452 mapwrite 0x597c14 thru 0x59f6ee (0x7adb bytes) 000166[0] 1790734826.560890 read 0x0b5e47 thru 0x0b874d (0x2907 bytes) 000167[1] 1790734826.613495 punch from 0x51405d to 0x51ebd5 (0xab79 bytes) 000170[1] 1790734826.654838 write 0x810cfd thru 0x815cd1 (0x4fd5 bytes) 000173[0] 1790734826.712721 mapread 0x3c9715 thru 0x3cd3da (0x3cc6 bytes) 000177[1] 1790734826.839556 read 0x6a8e08 thru 0x6a9178 (0x371 bytes) 000178[0] 1790734826.888799 mapwrite 0x8fd4c8 thru 0x90b374 (0xdead bytes) 000179[1] 1790734827.204871 read 0x83d90f thru 0x84986d (0xbf5f bytes) 000181[1] 1790734827.299614 mapwrite 0x66aa76 thru 0x676557 (0xbae2 bytes) 000182[0] 1790734827.440480 mapread 0x08fea8 thru 0x09a81a (0xa973 bytes) 000183[0] 1790734827.492086 mapwrite 0x4c7029 thru 0x4ce81d (0x77f5 bytes) 000184[0] 1790734827.606005 write 0x23ba52 thru 0x23be17 (0x3c6 bytes) 000186[0] 1790734827.633762 punch from 0x4689a8 to 0x46cde7 (0x4440 bytes) 000187[1] 1790734827.668702 trunc from 0x90b375 to 0x7d9de7 (0x13158d bytes) 000189[0] 1790734827.763472 read 0x678694 thru 0x686c5d (0xe5ca bytes) 000190[1] 1790734827.805360 mapread 0x402851 thru 0x41180d (0xefbd bytes) 000191[0] 1790734827.847860 mapwrite 0x38696b thru 0x38bc6a (0x5300 bytes) 000192[0] 1790734828.007009 mapwrite 0x3848ec thru 0x38cd3d (0x8452 bytes) 000193[0] 1790734828.115857 mapwrite 0x25e637 thru 0x26c509 (0xded3 bytes) 000194[0] 1790734828.256638 write 0x29a66a thru 0x2a7205 (0xcb9c bytes) 000195[0] 1790734828.268697 read 0x750985 thru 0x75f2dc (0xe958 bytes) 000197[1] 1790734828.294168 write 0x222b45 thru 0x22820d (0x56c9 bytes) 000199[1] 1790734828.361866 punch from 0x516f75 to 0x526e4f (0xfedb bytes) 000202[1] 1790734828.505283 trunc from 0x7d9de8 to 0x19d98b (0x63c45c bytes) 000203[0] 1790734828.585779 read 0x0b4464 thru 0x0c234f (0xdeec bytes) 000205[0] 1790734828.696858 write 0x7c8a4a thru 0x7cd388 (0x493f bytes) 000206[1] 1790734828.751201 write 0x3c9bee thru 0x3d9457 (0xf86a bytes) 000208[0] 1790734828.843034 mapread 0x364cf5 thru 0x366f69 (0x2275 bytes) 000213[1] 1790734829.082668 mapread 0x0076d0 thru 0x00c8a8 (0x51d9 bytes) 000219[0] 1790734829.225949 mapwrite 0x2199e9 thru 0x221f67 (0x857f bytes) 000220[0] 1790734829.421290 punch from 0x724313 to 0x7318fa (0xd5e8 bytes) 000221[0] 1790734829.471773 write 0x4c39b0 thru 0x4d1ce1 (0xe332 bytes) 000225[0] 1790734829.602519 punch from 0x5f7cbc to 0x601346 (0x968b bytes) 000227[0] 1790734829.661810 trunc from 0x7cd389 to 0x73cb8d (0x907fb bytes) 000229[0] 1790734829.840330 trunc from 0x73cb8e to 0x77bec8 (0x3f33b bytes) 000231[1] 1790734829.972473 read 0x25126d thru 0x2546bf (0x3453 bytes) 000235[1] 1790734830.011354 write 0x6153bb thru 0x61d026 (0x7c6c bytes) 000236[0] 1790734830.066058 mapwrite 0x4b0681 thru 0x4b1b06 (0x1486 bytes) 000239[0] 1790734830.394822 read 0x32e844 thru 0x3339ae (0x516b bytes) 000241[0] 1790734830.449812 trunc from 0x77bec9 to 0x07a8f9 (0x7015cf bytes) 000242[0] 1790734830.516146 read 0x011a6f thru 0x021311 (0xf8a3 bytes) 000243[1] 1790734830.532211 mapwrite 0x118c26 thru 0x11fabe (0x6e99 bytes) 000244[0] 1790734830.787656 trunc from 0x11fabf to 0x3781aa (0x2586ec bytes) 000246[0] 1790734830.882970 punch from 0x31e724 to 0x32da30 (0xf30d bytes) 000248[1] 1790734830.915890 mapwrite 0x75947c thru 0x76487d (0xb402 bytes) 000252[0] 1790734831.245734 mapread 0x3dc38b thru 0x3e41a4 (0x7e1a bytes) 000255[1] 1790734831.446357 read 0x2433f6 thru 0x24a470 (0x707b bytes) 000257[1] 1790734831.461225 trunc from 0x76487e to 0x57e404 (0x1e6479 bytes) 000258[1] 1790734831.513000 punch from 0x16f9d2 to 0x1770cf (0x76fe bytes) 000261[0] 1790734831.601309 read 0x09e896 thru 0x0a3039 (0x47a4 bytes) 000271[1] 1790734831.704363 trunc from 0x57e405 to 0x5bb200 (0x3cdfc bytes) 000272[0] 1790734831.745531 trunc from 0x5bb201 to 0x1caaa2 (0x3f075e bytes) 000273[0] 1790734831.870364 write 0x275fd7 thru 0x27d308 (0x7332 bytes) 000275[1] 1790734831.882163 punch from 0x05a8d2 to 0x0663ff (0xbb2e bytes) 000276[1] 1790734831.944456 mapwrite 0x0318a8 thru 0x03dee7 (0xc640 bytes) 000278[0] 1790734832.229494 write 0x34e9f0 thru 0x35c69a (0xdcab bytes) 000279[1] 1790734832.341216 mapread 0x101971 thru 0x105299 (0x3929 bytes) 000280[0] 1790734832.489179 mapread 0x231b9a thru 0x23e3c8 (0xc82f bytes) 000282[1] 1790734832.610474 punch from 0x1f461f to 0x1fd8eb (0x92cd bytes) 000285[0] 1790734832.744355 punch from 0x0aa789 to 0x0ab072 (0x8ea bytes) 000287[0] 1790734832.837773 read 0x21dcdb thru 0x22624f (0x8575 bytes) 000289[1] 1790734832.990707 mapread 0x267683 thru 0x26cd17 (0x5695 bytes) 000290[0] 1790734833.081397 punch from 0x2673a2 to 0x2744b6 (0xd115 bytes) 000293[1] 1790734833.130526 mapwrite 0x55f558 thru 0x565d82 (0x682b bytes) 000294[0] 1790734833.243289 mapwrite 0x715330 thru 0x71b71c (0x63ed bytes) 000296[0] 1790734833.576186 mapread 0x054d54 thru 0x05c8b0 (0x7b5d bytes) 000302[1] 1790734833.715671 mapread 0x1fe269 thru 0x207964 (0x96fc bytes) 000303[0] 1790734833.771811 punch from 0x000d02 to 0x01034f (0xf64e bytes) 000306[0] 1790734833.831601 mapwrite 0x6e5f14 thru 0x6f3d1a (0xde07 bytes) 000307[1] 1790734833.878311 write 0x1f62dd thru 0x1fb6bc (0x53e0 bytes) 000309[1] 1790734833.915844 punch from 0x4a1e1c to 0x4aeb79 (0xcd5e bytes) 000310[0] 1790734833.946316 read 0x466821 thru 0x466a69 (0x249 bytes) 000311[1] 1790734833.976684 read 0x0a9f78 thru 0x0abc05 (0x1c8e bytes) 000313[1] 1790734834.012527 punch from 0x3c24b5 to 0x3c9406 (0x6f52 bytes) 000314[1] 1790734834.022209 trunc from 0x71b71d to 0x60dd00 (0x10da1c bytes) 000317[0] 1790734834.105576 punch from 0x013218 to 0x01d474 (0xa25d bytes) 000318[1] 1790734834.145783 read 0x0c1f05 thru 0x0cb15f (0x925b bytes) 000319[1] 1790734834.196870 trunc from 0x60dd01 to 0x1b9c3c (0x4540c4 bytes) 000320[0] 1790734834.264320 write 0x365944 thru 0x36c171 (0x682e bytes) 000321[1] 1790734834.302661 write 0x4361bb thru 0x43eb48 (0x898e bytes) 000322[1] 1790734834.323656 write 0x8ab69f thru 0x8b6857 (0xb1b9 bytes) 000323[1] 1790734834.379757 mapread 0x2c80a5 thru 0x2ca161 (0x20bd bytes) 000324[1] 1790734834.405281 mapread 0x074f1f thru 0x07a0ca (0x51ac bytes) 000329[1] 1790734834.470382 punch from 0x38da3c to 0x38ea35 (0xffa bytes) 000331[0] 1790734834.478225 mapread 0x05232b thru 0x05c43c (0xa112 bytes) 000333[1] 1790734834.570352 punch from 0x8413e9 to 0x8506f9 (0xf311 bytes) 000334[1] 1790734834.623716 mapwrite 0x395818 thru 0x395cce (0x4b7 bytes) 000335[1] 1790734834.685591 trunc from 0x8b6858 to 0x070af0 (0x845d67 bytes) 000336[1] 1790734834.752954 write 0x4b8f14 thru 0x4bbf10 (0x2ffd bytes) 000338[0] 1790734834.850489 mapread 0x454eee thru 0x45748f (0x25a2 bytes) 000340[1] 1790734834.965644 trunc from 0x4bbf11 to 0x3cd554 (0xee9bc bytes) 000342[1] 1790734835.037578 read 0x22af4d thru 0x235bd6 (0xac8a bytes) 000343[0] 1790734835.152286 read 0x024897 thru 0x02e907 (0xa071 bytes) 000344[0] 1790734835.264129 trunc from 0x3cd555 to 0x5d89b0 (0x20b45c bytes) 000345[1] 1790734835.398815 mapread 0x3f80e4 thru 0x403cee (0xbc0b bytes) 000347[1] 1790734835.566676 mapwrite 0x121944 thru 0x1307df (0xee9c bytes) 000348[0] 1790734835.818053 read 0x1653f3 thru 0x16ad4c (0x595a bytes) 000349[1] 1790734835.849791 mapwrite 0x9c9916 thru 0x9d5570 (0xbc5b bytes) 000350[0] 1790734836.041594 trunc from 0x9d5571 to 0x7bd8c6 (0x217caa bytes) 000352[0] 1790734836.094404 trunc from 0x7bd8c7 to 0x0de480 (0x6df446 bytes) 000353[1] 1790734836.124392 mapread 0x0ae4a4 thru 0x0ae517 (0x74 bytes) 000354[0] 1790734836.162107 write 0x3b4cef thru 0x3c20ac (0xd3be bytes) 000356[1] 1790734836.236871 mapwrite 0x554e81 thru 0x564e3e (0xffbe bytes) 000359[1] 1790734836.450153 punch from 0x080660 to 0x08219c (0x1b3d bytes) 000360[1] 1790734836.461221 trunc from 0x564e3f to 0x949760 (0x3e4922 bytes) 000361[1] 1790734836.502365 punch from 0x03df24 to 0x04b72e (0xd80b bytes) 000363[1] 1790734836.519400 mapwrite 0x2e76e8 thru 0x2ee997 (0x72b0 bytes) 000364[0] 1790734836.780673 read 0x7de42e thru 0x7df53a (0x110d bytes) 000366[0] 1790734836.811497 mapwrite 0x4b1aae thru 0x4bf0fa (0xd64d bytes) 000367[1] 1790734836.925373 read 0x1095e5 thru 0x11702e (0xda4a bytes) 000368[0] 1790734836.950143 mapread 0x769b98 thru 0x776630 (0xca99 bytes) 000370[0] 1790734837.014920 mapwrite 0x4fc2f0 thru 0x509924 (0xd635 bytes) 000371[0] 1790734837.107334 write 0x6aa675 thru 0x6ac1ce (0x1b5a bytes) 000372[0] 1790734837.137633 write 0x6e30bd thru 0x6e68e8 (0x382c bytes) 000375[1] 1790734837.194961 write 0x344a13 thru 0x35193e (0xcf2c bytes) 000378[0] 1790734837.251738 punch from 0x6d724f to 0x6dec2b (0x79dd bytes) 000379[0] 1790734837.277356 read 0x81e92e thru 0x822e84 (0x4557 bytes) 000384[0] 1790734837.337634 punch from 0x452e54 to 0x4570fe (0x42ab bytes) 000385[0] 1790734837.343506 trunc from 0x949761 to 0x873eff (0xd5861 bytes) 000386[1] 1790734837.413676 mapwrite 0x70e8b2 thru 0x70fb5e (0x12ad bytes) 000387[0] 1790734837.645279 mapread 0x5b5352 thru 0x5bdd72 (0x8a21 bytes) 000389[1] 1790734837.853989 read 0x62203e thru 0x631b4e (0xfb11 bytes) 000390[1] 1790734837.886543 mapwrite 0x9ffe91 thru 0x9fffff (0x16f bytes) 000393[1] 1790734838.206265 punch from 0x1b8498 to 0x1bcaa0 (0x4609 bytes) 000394[1] 1790734838.268825 write 0x2276ce thru 0x23252c (0xae5f bytes) 000395[0] 1790734838.315011 mapread 0x756ae4 thru 0x760e2e (0xa34b bytes) 000401[0] 1790734838.513241 mapwrite 0x946dd7 thru 0x95315d (0xc387 bytes) 000402[1] 1790734838.713085 mapread 0x4c1634 thru 0x4c29f6 (0x13c3 bytes) 000404[1] 1790734838.796074 read 0x2a5e1d thru 0x2a86a0 (0x2884 bytes) 000409[0] 1790734838.834356 mapread 0x85f475 thru 0x86476b (0x52f7 bytes) 000410[1] 1790734838.854139 punch from 0x69bc92 to 0x6a79dd (0xbd4c bytes) 000411[0] 1790734838.879520 punch from 0x16c59f to 0x173c88 (0x76ea bytes) 000412[1] 1790734838.919726 write 0x891da4 thru 0x898f00 (0x715d bytes) 000413[1] 1790734838.973117 mapwrite 0x526dc6 thru 0x534c9e (0xded9 bytes) 000416[1] 1790734839.158811 trunc from 0xa00000 to 0x3b5a3f (0x64a5c0 bytes) 000418[0] 1790734839.232799 write 0x5917c9 thru 0x5a1260 (0xfa98 bytes) 000419[1] 1790734839.268187 trunc from 0x5a1261 to 0x2cc5f6 (0x2d4c6a bytes) 000420[0] 1790734839.327559 mapwrite 0x47d31e thru 0x487c69 (0xa94c bytes) 000421[0] 1790734839.493262 read 0x463f7c thru 0x465ad6 (0x1b5b bytes) 000422[0] 1790734839.504640 write 0x476f8b thru 0x4771b1 (0x227 bytes) 000426[1] 1790734839.588843 trunc from 0x487c6a to 0x0ba3be (0x3cd8ab bytes) 000427[1] 1790734839.666176 mapread 0x06ad2b thru 0x072274 (0x754a bytes) 000428[1] 1790734839.696403 mapread 0x070b22 thru 0x0723a7 (0x1886 bytes) 000431[0] 1790734839.793558 mapread 0x03d199 thru 0x045fea (0x8e52 bytes) 000432[0] 1790734839.888557 write 0x6f07e5 thru 0x6f24cd (0x1ce9 bytes) 000433[1] 1790734839.955347 punch from 0x501228 to 0x50d8c0 (0xc699 bytes) 000434[0] 1790734840.023408 mapread 0x1b7b11 thru 0x1c774d (0xfc3d bytes) 000435[1] 1790734840.283576 write 0x37598d thru 0x384790 (0xee04 bytes) 000436[1] 1790734840.360263 mapread 0x047f64 thru 0x04efa1 (0x703e bytes) 000437[1] 1790734840.373929 punch from 0x64f628 to 0x656bbd (0x7596 bytes) 000439[1] 1790734840.428113 read 0x187d56 thru 0x19507a (0xd325 bytes) 000440[0] 1790734840.456563 mapread 0x1eb74a thru 0x1f1c6b (0x6522 bytes) 000441[0] 1790734840.509975 punch from 0x2ce652 to 0x2dcb54 (0xe503 bytes) 000444[1] 1790734840.623471 mapread 0x585578 thru 0x58dbd6 (0x865f bytes) 000445[0] 1790734840.710378 read 0x1aa47f thru 0x1aaaac (0x62e bytes) 000447[0] 1790734840.769129 punch from 0x5c9b04 to 0x5d8724 (0xec21 bytes) 000448[1] 1790734840.803664 mapread 0x1e20f7 thru 0x1e7312 (0x521c bytes) 000449[0] 1790734840.918222 trunc from 0x6f24ce to 0x527761 (0x1cad6c bytes) 000450[0] 1790734841.056647 punch from 0x521d95 to 0x527761 (0x59cd bytes) 000451[0] 1790734841.153161 write 0x67f2d2 thru 0x6833ad (0x40dc bytes) 000452[0] 1790734841.165183 mapread 0x5debe0 thru 0x5ea2de (0xb6ff bytes) 000454[1] 1790734841.280145 read 0x2509ee thru 0x2542c2 (0x38d5 bytes) 000455[0] 1790734841.408233 punch from 0x1ddbec to 0x1e8336 (0xa74b bytes) 000456[0] 1790734841.461971 trunc from 0x6833ae to 0x2a82df (0x3db0ce bytes) 000458[1] 1790734841.637804 mapread 0x240eac thru 0x24c24b (0xb3a0 bytes) 000460[1] 1790734841.835197 write 0x9560c8 thru 0x96070a (0xa643 bytes) 000463[1] 1790734842.070546 mapread 0x389f1b thru 0x397d83 (0xde69 bytes) 000464[1] 1790734842.194473 mapread 0x60be73 thru 0x61260b (0x6799 bytes) 000465[0] 1790734842.291923 mapread 0x5b8b57 thru 0x5c1441 (0x88eb bytes) 000466[0] 1790734842.616043 mapwrite 0x86bcbb thru 0x872d3c (0x7082 bytes) 000468[1] 1790734842.883966 write 0x5b79c8 thru 0x5b7d01 (0x33a bytes) 000470[0] 1790734842.971011 trunc from 0x96070b to 0x364813 (0x5fbef7 bytes) 000471[1] 1790734843.056103 trunc from 0x364814 to 0x18cca1 (0x1d7b72 bytes) 000472[0] 1790734843.167487 mapread 0x0a0de9 thru 0x0afb3e (0xed56 bytes) 000474[0] 1790734843.263395 punch from 0x04b55f to 0x0538b3 (0x8355 bytes) 000476[1] 1790734843.312750 mapread 0x1378c0 thru 0x146e1d (0xf55e bytes) 000479[0] 1790734843.401712 write 0x1be454 thru 0x1c7fc6 (0x9b73 bytes) 000480[1] 1790734843.442371 mapwrite 0x41734c thru 0x426d64 (0xfa19 bytes) 000483[1] 1790734843.720597 write 0x324824 thru 0x32d2a3 (0x8a80 bytes) 000484[1] 1790734843.816165 mapread 0x20a4d2 thru 0x20eb50 (0x467f bytes) 000485[1] 1790734843.911491 mapread 0x0db9c2 thru 0x0df1df (0x381e bytes) 000486[1] 1790734843.958910 read 0x289e39 thru 0x293d68 (0x9f30 bytes) 000487[1] 1790734844.038320 mapread 0x1d02b3 thru 0x1dcc90 (0xc9de bytes) 000488[1] 1790734844.067393 trunc from 0x426d65 to 0x07a1f9 (0x3acb6b bytes) 000490[1] 1790734844.187298 punch from 0x014067 to 0x0205e1 (0xc57b bytes) 000491[1] 1790734844.222965 write 0x816794 thru 0x816b38 (0x3a5 bytes) 000492[1] 1790734844.229483 trunc from 0x816b39 to 0x630d63 (0x1e5dd5 bytes) 000493[1] 1790734844.300249 trunc from 0x630d64 to 0x295909 (0x39b45a bytes) 000494[1] 1790734844.372277 mapread 0x0f2b58 thru 0x0f8944 (0x5ded bytes) 000497[1] 1790734844.519068 write 0x33c178 thru 0x3446e6 (0x856f bytes) 000498[0] 1790734844.547376 punch from 0x010b68 to 0x010fde (0x477 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4844 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000006[1] 1790734845.681634 mapwrite 0x0c8000 thru 0x0d23a3 (0xa3a4 bytes) 000007[0] 1790734845.938472 write 0x72b000 thru 0x738ad6 (0xdad7 bytes) 000008[0] 1790734845.985797 punch from 0x08cfdf to 0x091980 (0x49a2 bytes) 000012[0] 1790734846.074307 write 0x5b1000 thru 0x5bd302 (0xc303 bytes) 000014[1] 1790734846.126010 punch from 0x67d7c3 to 0x689aca (0xc308 bytes) 000017[1] 1790734846.192222 write 0x802000 thru 0x80e046 (0xc047 bytes) 000018[1] 1790734846.194065 punch from 0x3370cb to 0x33fbed (0x8b23 bytes) 000019[0] 1790734846.207253 write 0x991000 thru 0x998c9f (0x7ca0 bytes) 000022[0] 1790734846.240059 read 0x51a000 thru 0x529275 (0xf276 bytes) 000023[0] 1790734846.250397 read 0x406000 thru 0x408432 (0x2433 bytes) 000024[0] 1790734846.255692 punch from 0x26bae1 to 0x270c31 (0x5151 bytes) truncating to largest ever: 0x77161f 000025[0] 1790734846.259213 trunc from 0x998ca0 to 0x77161e (0x227681 bytes) 000027[1] 1790734846.274114 read 0x06a000 thru 0x074718 (0xa719 bytes) 000028[0] 1790734846.298554 write 0x961000 thru 0x961186 (0x187 bytes) 000031[1] 1790734846.341523 punch from 0x1dc54d to 0x1e57fe (0x92b2 bytes) 000032[1] 1790734846.360104 mapread 0x361000 thru 0x366993 (0x5994 bytes) 000033[0] 1790734846.384297 write 0x55e000 thru 0x568be1 (0xabe2 bytes) 000034[0] 1790734846.400404 punch from 0x0ffc81 to 0x1073d7 (0x7757 bytes) 000036[1] 1790734846.424242 write 0x9de000 thru 0x9e1de0 (0x3de1 bytes) truncating to largest ever: 0x861c99 000037[1] 1790734846.449375 trunc from 0x9e1de1 to 0x861c98 (0x180148 bytes) 000039[1] 1790734846.476959 read 0x0d9000 thru 0x0e6fda (0xdfdb bytes) 000040[0] 1790734846.497509 mapread 0x402000 thru 0x403a77 (0x1a78 bytes) 000041[1] 1790734846.535543 write 0x79b000 thru 0x7a52b6 (0xa2b7 bytes) 000043[1] 1790734846.567264 punch from 0x1c9e3a to 0x1cf37c (0x5543 bytes) 000044[0] 1790734846.583075 read 0x3c8000 thru 0x3ccd01 (0x4d02 bytes) 000046[1] 1790734846.603478 mapwrite 0x79c000 thru 0x7a0c44 (0x4c45 bytes) 000047[0] 1790734846.670687 mapwrite 0x7f8000 thru 0x800905 (0x8906 bytes) 000049[0] 1790734846.765210 read 0x315000 thru 0x31a5d8 (0x55d9 bytes) 000052[1] 1790734846.784139 write 0x4e3000 thru 0x4f25df (0xf5e0 bytes) 000053[1] 1790734846.806390 read 0x4dd000 thru 0x4e82f6 (0xb2f7 bytes) 000054[0] 1790734846.813277 mapread 0x091000 thru 0x09274b (0x174c bytes) 000055[1] 1790734846.847235 trunc from 0x861c99 to 0x08a0f5 (0x7d7ba3 bytes) 000056[0] 1790734846.873006 write 0x504000 thru 0x50e6aa (0xa6ab bytes) 000061[0] 1790734846.918580 trunc from 0x50e6ab to 0x3e6afe (0x127bac bytes) 000064[0] 1790734846.952090 write 0x5e9000 thru 0x5eb48c (0x248d bytes) 000065[0] 1790734846.967897 punch from 0x136981 to 0x137866 (0xee6 bytes) 000066[1] 1790734846.971700 mapwrite 0x052000 thru 0x052af1 (0xaf2 bytes) 000068[0] 1790734847.094149 mapwrite 0x047000 thru 0x05135c (0xa35d bytes) 000070[1] 1790734847.222381 trunc from 0x5eb48d to 0x6b4cf8 (0xc986c bytes) 000071[0] 1790734847.241990 mapwrite 0x63a000 thru 0x640b0b (0x6b0c bytes) 000072[1] 1790734847.322356 trunc from 0x6b4cf9 to 0x397a4e (0x31d2aa bytes) 000073[0] 1790734847.356710 mapwrite 0x118000 thru 0x123f56 (0xbf57 bytes) 000075[1] 1790734847.480508 mapread 0x333000 thru 0x339b01 (0x6b02 bytes) 000076[0] 1790734847.517584 write 0x428000 thru 0x42f71a (0x771b bytes) 000077[1] 1790734847.545734 punch from 0x236783 to 0x240f67 (0xa7e5 bytes) 000079[0] 1790734847.578970 read 0x42c000 thru 0x42f4c7 (0x34c8 bytes) 000080[1] 1790734847.600510 write 0x82a000 thru 0x833a5b (0x9a5c bytes) 000083[1] 1790734847.617911 trunc from 0x833a5c to 0x72e64d (0x10540e bytes) 000084[0] 1790734847.637763 punch from 0x4ba0d0 to 0x4bc113 (0x2044 bytes) 000088[1] 1790734847.678267 trunc from 0x72e64e to 0x334487 (0x3fa1c6 bytes) 000089[1] 1790734847.699170 mapread 0x114000 thru 0x11f991 (0xb992 bytes) truncating to largest ever: 0x8c3449 000090[0] 1790734847.732532 trunc from 0x334488 to 0x8c3448 (0x58efc1 bytes) 000091[1] 1790734847.770458 trunc from 0x8c3449 to 0x187979 (0x73bacf bytes) 000092[1] 1790734847.806353 mapread 0x126000 thru 0x129de6 (0x3de7 bytes) 000093[1] 1790734847.835528 punch from 0x0a8f65 to 0x0b87a9 (0xf845 bytes) 000094[0] 1790734847.860835 read 0x10a000 thru 0x11619c (0xc19d bytes) truncating to largest ever: 0x99ad2a 000095[1] 1790734847.922069 trunc from 0x18797a to 0x99ad29 (0x8133b0 bytes) 000097[0] 1790734847.995950 write 0x1a6000 thru 0x1b14b0 (0xb4b1 bytes) 000099[1] 1790734848.040165 mapread 0x08d000 thru 0x08db21 (0xb22 bytes) 000101[1] 1790734848.079148 punch from 0x763629 to 0x76775d (0x4135 bytes) 000102[1] 1790734848.095254 trunc from 0x99ad2a to 0x3479ca (0x65335f bytes) 000103[0] 1790734848.117963 trunc from 0x3479cb to 0x719ecb (0x3d2501 bytes) 000104[0] 1790734848.150824 mapread 0x55f000 thru 0x560d4c (0x1d4d bytes) 000106[0] 1790734848.250315 trunc from 0x719ecc to 0x22d1a1 (0x4ecd2a bytes) 000107[1] 1790734848.277316 trunc from 0x22d1a2 to 0x1a2030 (0x8b171 bytes) 000108[1] 1790734848.362860 trunc from 0x1a2031 to 0x45be6c (0x2b9e3c bytes) 000109[0] 1790734848.424729 mapread 0x22c000 thru 0x236d6e (0xad6f bytes) 000110[1] 1790734848.548974 punch from 0x156d9f to 0x15a693 (0x38f5 bytes) 000111[1] 1790734848.608121 trunc from 0x45be6d to 0x578ab3 (0x11cc47 bytes) 000112[1] 1790734848.642148 read 0x267000 thru 0x26f8d0 (0x88d1 bytes) 000113[1] 1790734848.751867 write 0x479000 thru 0x4879aa (0xe9ab bytes) 000115[1] 1790734848.790715 punch from 0x31826b to 0x31dd34 (0x5aca bytes) 000116[0] 1790734848.794940 mapwrite 0x330000 thru 0x337a91 (0x7a92 bytes) 000119[0] 1790734848.978488 write 0x368000 thru 0x376e16 (0xee17 bytes) 000120[0] 1790734849.022110 mapread 0x1d1000 thru 0x1d56de (0x46df bytes) 000123[1] 1790734849.075330 punch from 0x5603d3 to 0x56fde9 (0xfa17 bytes) 000124[0] 1790734849.153737 punch from 0x178c6e to 0x18443b (0xb7ce bytes) 000125[1] 1790734849.207147 read 0x1c8000 thru 0x1d7a32 (0xfa33 bytes) 000126[0] 1790734849.275551 write 0x4e8000 thru 0x4f2161 (0xa162 bytes) 000127[1] 1790734849.341316 punch from 0x1c3a51 to 0x1c3cfa (0x2aa bytes) 000130[1] 1790734849.424334 mapwrite 0x5d8000 thru 0x5e2fff (0xb000 bytes) 000131[0] 1790734849.761268 mapwrite 0x81e000 thru 0x827c23 (0x9c24 bytes) 000134[0] 1790734850.209952 punch from 0x03d839 to 0x0486de (0xaea6 bytes) 000135[0] 1790734850.263802 mapwrite 0x969000 thru 0x96dd29 (0x4d2a bytes) 000137[1] 1790734850.506265 trunc from 0x96dd2a to 0x90e2b6 (0x5fa73 bytes) 000140[0] 1790734850.636921 mapwrite 0x905000 thru 0x9081e5 (0x31e6 bytes) 000141[0] 1790734850.736016 mapwrite 0x595000 thru 0x5a0f06 (0xbf07 bytes) 000142[0] 1790734851.147151 mapread 0x19a000 thru 0x1a2715 (0x8716 bytes) 000143[1] 1790734851.235308 mapread 0x277000 thru 0x278b1c (0x1b1d bytes) 000144[0] 1790734851.309004 mapwrite 0x8ee000 thru 0x8fdf1a (0xff1b bytes) 000147[1] 1790734851.511578 mapread 0x0c7000 thru 0x0cc3cf (0x53d0 bytes) 000148[1] 1790734851.566535 mapwrite 0x8a0000 thru 0x8ac103 (0xc104 bytes) 000149[0] 1790734851.781598 mapwrite 0x6ac000 thru 0x6b209c (0x609d bytes) 000151[1] 1790734852.000006 mapwrite 0x833000 thru 0x8429d5 (0xf9d6 bytes) 000152[0] 1790734852.103934 mapwrite 0x7ae000 thru 0x7b15ff (0x3600 bytes) 000156[1] 1790734852.236782 trunc from 0x90e2b7 to 0x3f6168 (0x51814e bytes) 000157[0] 1790734852.287150 read 0x397000 thru 0x39ab1f (0x3b20 bytes) 000159[0] 1790734852.373967 write 0x2af000 thru 0x2be6d9 (0xf6da bytes) 000160[0] 1790734852.422893 trunc from 0x3f6169 to 0x67b9a7 (0x28583f bytes) 000161[0] 1790734852.444775 write 0x167000 thru 0x16b9cd (0x49ce bytes) 000162[0] 1790734852.481087 punch from 0x40a655 to 0x411366 (0x6d12 bytes) 000163[1] 1790734852.487343 mapwrite 0x25b000 thru 0x267db6 (0xcdb7 bytes) 000164[0] 1790734852.615101 trunc from 0x67b9a8 to 0x4a2437 (0x1d9570 bytes) 000165[0] 1790734852.661385 trunc from 0x4a2438 to 0x6a6a29 (0x2045f2 bytes) 000170[0] 1790734852.702195 read 0x43b000 thru 0x43e22c (0x322d bytes) 000172[1] 1790734852.725262 mapread 0x0de000 thru 0x0e982b (0xb82c bytes) 000178[1] 1790734852.809181 mapread 0x282000 thru 0x28283d (0x83e bytes) 000179[1] 1790734852.825001 write 0x3dd000 thru 0x3e6017 (0x9018 bytes) 000180[0] 1790734852.880259 mapwrite 0x342000 thru 0x34609c (0x409d bytes) 000182[0] 1790734853.067097 write 0x1c3000 thru 0x1cb7a5 (0x87a6 bytes) 000183[1] 1790734853.099263 write 0x26e000 thru 0x26e2bd (0x2be bytes) 000187[1] 1790734853.211057 mapwrite 0x4ec000 thru 0x4f200c (0x600d bytes) 000188[0] 1790734853.322168 mapwrite 0x55d000 thru 0x56c600 (0xf601 bytes) 000191[0] 1790734853.486475 mapread 0x3bf000 thru 0x3c0777 (0x1778 bytes) 000192[1] 1790734853.502033 mapread 0x2b8000 thru 0x2bfe66 (0x7e67 bytes) 000194[1] 1790734853.551781 read 0x62c000 thru 0x63a2a1 (0xe2a2 bytes) 000195[0] 1790734853.568318 mapwrite 0x7e9000 thru 0x7ef651 (0x6652 bytes) 000196[1] 1790734853.666949 mapwrite 0x864000 thru 0x86885e (0x485f bytes) 000198[0] 1790734853.803180 read 0x4f5000 thru 0x4f99a3 (0x49a4 bytes) 000202[0] 1790734853.825101 write 0x834000 thru 0x834184 (0x185 bytes) 000204[0] 1790734853.859082 mapread 0x586000 thru 0x5892e9 (0x32ea bytes) 000205[1] 1790734853.883319 mapread 0x798000 thru 0x7a5569 (0xd56a bytes) 000206[0] 1790734853.924789 trunc from 0x86885f to 0x2fb1cb (0x56d693 bytes) 000207[0] 1790734853.962090 punch from 0x0d2c43 to 0x0dc257 (0x9615 bytes) 000208[0] 1790734853.970203 mapwrite 0x743000 thru 0x743346 (0x347 bytes) 000210[0] 1790734854.022479 read 0x4a9000 thru 0x4b7e44 (0xee45 bytes) 000212[0] 1790734854.084352 read 0x6a7000 thru 0x6ac72b (0x572c bytes) 000213[0] 1790734854.103287 mapread 0x27e000 thru 0x282be1 (0x4be2 bytes) 000214[0] 1790734854.118714 write 0x4dd000 thru 0x4ec211 (0xf212 bytes) 000215[1] 1790734854.133752 punch from 0x633293 to 0x63991a (0x6688 bytes) 000217[0] 1790734854.167836 mapread 0x1fd000 thru 0x20a5ec (0xd5ed bytes) 000218[0] 1790734854.210789 write 0x2ea000 thru 0x2f0d05 (0x6d06 bytes) 000219[0] 1790734854.221758 mapread 0x357000 thru 0x366225 (0xf226 bytes) 000220[0] 1790734854.243796 read 0x113000 thru 0x113bee (0xbef bytes) 000226[1] 1790734854.351042 mapwrite 0x5ad000 thru 0x5bc5e6 (0xf5e7 bytes) 000227[1] 1790734854.509209 write 0x9a2000 thru 0x9a704d (0x504e bytes) 000228[0] 1790734854.545519 trunc from 0x9a704e to 0x75f553 (0x247afa bytes) 000229[1] 1790734854.669589 write 0x898000 thru 0x8a3a66 (0xba67 bytes) 000230[1] 1790734854.728667 read 0x5bb000 thru 0x5c2659 (0x765a bytes) 000234[1] 1790734854.845977 read 0x860000 thru 0x86b4bc (0xb4bd bytes) 000235[0] 1790734854.854077 punch from 0x7bd611 to 0x7cc0cb (0xeabb bytes) 000236[1] 1790734854.923935 write 0x8fa000 thru 0x9091e1 (0xf1e2 bytes) 000237[1] 1790734854.964015 trunc from 0x9091e2 to 0x22e5e3 (0x6dabfe bytes) 000239[1] 1790734855.073362 mapwrite 0x209000 thru 0x2108d3 (0x78d4 bytes) 000240[1] 1790734855.212749 trunc from 0x22e5e4 to 0x074246 (0x1ba39d bytes) 000241[0] 1790734855.278478 trunc from 0x074247 to 0x222c6a (0x1aea24 bytes) 000242[1] 1790734855.322363 trunc from 0x222c6b to 0x9741af (0x751545 bytes) 000243[0] 1790734855.382400 write 0x3e7000 thru 0x3eb8de (0x48df bytes) 000244[0] 1790734855.406722 read 0x63e000 thru 0x63f70c (0x170d bytes) 000245[1] 1790734855.415164 trunc from 0x9741b0 to 0x0aac23 (0x8c958c bytes) 000246[0] 1790734855.457340 mapread 0x0a6000 thru 0x0aa850 (0x4851 bytes) 000248[0] 1790734855.602367 punch from 0x087b71 to 0x08c19a (0x462a bytes) 000249[1] 1790734855.650977 mapread 0x05e000 thru 0x065e83 (0x7e84 bytes) 000252[1] 1790734855.731787 write 0x711000 thru 0x7189c6 (0x79c7 bytes) 000253[1] 1790734855.761981 mapread 0x6ed000 thru 0x6f0dc3 (0x3dc4 bytes) 000254[1] 1790734855.840841 write 0x8e9000 thru 0x8f5f63 (0xcf64 bytes) 000260[1] 1790734855.880562 trunc from 0x8f5f64 to 0x38f043 (0x566f20 bytes) 000261[0] 1790734855.903952 trunc from 0x38f044 to 0x66c9ef (0x2dd9ac bytes) 000262[1] 1790734855.974606 mapread 0x5b9000 thru 0x5bcbe6 (0x3be7 bytes) 000263[0] 1790734856.029243 mapwrite 0x938000 thru 0x938574 (0x575 bytes) 000264[0] 1790734856.184284 trunc from 0x938575 to 0x826ffe (0x111576 bytes) 000266[1] 1790734856.212921 mapread 0x6b7000 thru 0x6c5d0e (0xed0f bytes) 000267[1] 1790734856.258285 punch from 0x7b78b9 to 0x7c176e (0x9eb6 bytes) 000268[0] 1790734856.278222 write 0x8e1000 thru 0x8ef9a8 (0xe9a9 bytes) 000269[0] 1790734856.307004 write 0x553000 thru 0x555d45 (0x2d46 bytes) 000271[1] 1790734856.319370 read 0x5a7000 thru 0x5b2e3a (0xbe3b bytes) 000273[1] 1790734856.364436 mapread 0x7ea000 thru 0x7ea046 (0x47 bytes) 000274[0] 1790734856.373958 punch from 0x7128bf to 0x719993 (0x70d5 bytes) 000275[1] 1790734856.424563 read 0x514000 thru 0x51d101 (0x9102 bytes) 000278[1] 1790734856.456889 punch from 0x64e57f to 0x656bed (0x866f bytes) 000279[0] 1790734856.490770 punch from 0x2b8518 to 0x2c0357 (0x7e40 bytes) 000280[1] 1790734856.537080 punch from 0x89b5ed to 0x8a2da7 (0x77bb bytes) 000281[0] 1790734856.559135 punch from 0x528598 to 0x5336cc (0xb135 bytes) 000285[1] 1790734856.601091 punch from 0x692f1b to 0x69353a (0x620 bytes) 000286[1] 1790734856.615701 mapread 0x65a000 thru 0x669e7a (0xfe7b bytes) 000289[1] 1790734856.659686 mapread 0x19b000 thru 0x19ebc6 (0x3bc7 bytes) 000290[0] 1790734856.698782 mapwrite 0x9e8000 thru 0x9ec846 (0x4847 bytes) 000294[0] 1790734856.835590 mapwrite 0x8cb000 thru 0x8ceff7 (0x3ff8 bytes) 000295[0] 1790734856.923403 read 0x620000 thru 0x62de03 (0xde04 bytes) 000296[1] 1790734856.940957 mapread 0x200000 thru 0x20e589 (0xe58a bytes) 000297[0] 1790734856.983968 mapread 0x274000 thru 0x283101 (0xf102 bytes) 000300[0] 1790734857.098743 write 0x4e6000 thru 0x4eb201 (0x5202 bytes) 000301[0] 1790734857.215014 trunc from 0x9ec847 to 0x35f966 (0x68cee0 bytes) 000303[0] 1790734857.294269 mapwrite 0x001000 thru 0x007e2e (0x6e2f bytes) 000306[1] 1790734857.499664 write 0x310000 thru 0x319ed0 (0x9ed1 bytes) 000308[1] 1790734857.532530 read 0x126000 thru 0x12e547 (0x8548 bytes) 000310[1] 1790734857.716298 punch from 0x1acdfb to 0x1b6279 (0x947f bytes) 000311[1] 1790734857.721143 punch from 0x31b63a to 0x31eeb5 (0x387c bytes) 000312[0] 1790734857.752898 write 0x5aa000 thru 0x5ae411 (0x4412 bytes) 000313[0] 1790734857.802965 mapwrite 0x97b000 thru 0x986a3f (0xba40 bytes) 000314[1] 1790734857.980359 write 0x026000 thru 0x0307fd (0xa7fe bytes) 000315[1] 1790734858.028062 read 0x0c7000 thru 0x0c89fe (0x19ff bytes) 000316[1] 1790734858.083190 mapwrite 0x2d9000 thru 0x2daca3 (0x1ca4 bytes) 000317[0] 1790734858.189778 read 0x340000 thru 0x340573 (0x574 bytes) 000318[0] 1790734858.223087 punch from 0x8eec90 to 0x8f7481 (0x87f2 bytes) 000321[1] 1790734858.319774 mapwrite 0x3ad000 thru 0x3ad430 (0x431 bytes) truncating to largest ever: 0x9e85ae 000322[0] 1790734858.545380 trunc from 0x986a40 to 0x9e85ad (0x61b6e bytes) 000325[0] 1790734858.636094 write 0x4c1000 thru 0x4cc02c (0xb02d bytes) 000326[1] 1790734858.674980 mapread 0x06b000 thru 0x06c5b6 (0x15b7 bytes) 000328[1] 1790734858.743646 mapread 0x789000 thru 0x78b08c (0x208d bytes) 000331[0] 1790734858.779817 mapread 0x9ad000 thru 0x9b3cda (0x6cdb bytes) 000333[1] 1790734858.813938 read 0x1a1000 thru 0x1a270a (0x170b bytes) 000334[0] 1790734858.820992 mapread 0x1d9000 thru 0x1d919d (0x19e bytes) 000337[1] 1790734858.829182 write 0x049000 thru 0x04b576 (0x2577 bytes) 000339[0] 1790734858.851951 read 0x7f2000 thru 0x7fc6b3 (0xa6b4 bytes) 000340[1] 1790734858.891239 mapwrite 0x718000 thru 0x71e7dc (0x67dd bytes) 000342[1] 1790734859.041168 punch from 0x5d385a to 0x5d462b (0xdd2 bytes) 000343[1] 1790734859.058225 trunc from 0x9e85ae to 0x31e73f (0x6c9e6e bytes) 000345[0] 1790734859.107845 write 0x7e4000 thru 0x7eef06 (0xaf07 bytes) 000346[0] 1790734859.141829 punch from 0x2bc08e to 0x2c8300 (0xc273 bytes) 000347[1] 1790734859.167501 trunc from 0x7eef07 to 0x4b85d5 (0x336931 bytes) 000348[0] 1790734859.218880 punch from 0x3753cf to 0x376888 (0x14ba bytes) 000350[0] 1790734859.263553 mapwrite 0x1e7000 thru 0x1f2c42 (0xbc43 bytes) 000351[0] 1790734859.381565 punch from 0x07486a to 0x07fc26 (0xb3bd bytes) 000353[0] 1790734859.406858 punch from 0x3060a7 to 0x311961 (0xb8bb bytes) 000359[0] 1790734859.499314 read 0x1e8000 thru 0x1ecc7b (0x4c7c bytes) 000360[1] 1790734859.500297 write 0x2e4000 thru 0x2e5933 (0x1934 bytes) 000361[0] 1790734859.534271 mapread 0x28e000 thru 0x2940a4 (0x60a5 bytes) 000362[0] 1790734859.575188 punch from 0x3cf941 to 0x3d830b (0x89cb bytes) 000363[1] 1790734859.609077 trunc from 0x4b85d6 to 0x64f0cf (0x196afa bytes) 000364[0] 1790734859.660753 mapread 0x5de000 thru 0x5dffdc (0x1fdd bytes) 000366[0] 1790734859.697194 mapwrite 0x7b0000 thru 0x7ba86d (0xa86e bytes) 000367[1] 1790734859.905050 punch from 0x20562a to 0x207d8e (0x2765 bytes) 000368[0] 1790734859.934279 punch from 0x614c51 to 0x61b6ce (0x6a7e bytes) 000369[1] 1790734859.978094 mapwrite 0x685000 thru 0x691585 (0xc586 bytes) 000370[0] 1790734860.089500 punch from 0x33e928 to 0x33f3e3 (0xabc bytes) 000372[0] 1790734860.131351 trunc from 0x7ba86e to 0x1ab0d4 (0x60f799 bytes) 000373[0] 1790734860.159215 read 0x131000 thru 0x13e545 (0xd546 bytes) 000378[1] 1790734860.279012 write 0x06b000 thru 0x076232 (0xb233 bytes) 000380[1] 1790734860.354351 punch from 0x027659 to 0x0313ff (0x9da7 bytes) 000382[0] 1790734860.368317 punch from 0x1941dd to 0x19bf4a (0x7d6e bytes) 000383[1] 1790734860.398762 write 0x8a8000 thru 0x8b6093 (0xe094 bytes) 000385[0] 1790734860.423590 punch from 0x40c658 to 0x413bd5 (0x757e bytes) 000386[1] 1790734860.485526 punch from 0x120ac7 to 0x128f65 (0x849f bytes) 000387[1] 1790734860.548233 trunc from 0x8b6094 to 0x0bada2 (0x7fb2f1 bytes) 000390[1] 1790734860.613564 trunc from 0x0bada3 to 0x1a1662 (0xe68c0 bytes) 000391[0] 1790734860.649317 write 0x877000 thru 0x8776cc (0x6cd bytes) 000397[1] 1790734860.702014 mapread 0x393000 thru 0x39fd70 (0xcd71 bytes) 000400[1] 1790734860.765336 mapwrite 0x8db000 thru 0x8e0f5c (0x5f5d bytes) 000401[1] 1790734860.869811 mapwrite 0x728000 thru 0x7338a9 (0xb8aa bytes) 000402[0] 1790734860.991685 write 0x538000 thru 0x53ec10 (0x6c11 bytes) 000403[0] 1790734861.051203 trunc from 0x8e0f5d to 0x232e49 (0x6ae113 bytes) 000406[1] 1790734861.100031 trunc from 0x232e4a to 0x136936 (0xfc513 bytes) 000408[0] 1790734861.159318 mapwrite 0x8ef000 thru 0x8fc615 (0xd616 bytes) 000409[0] 1790734861.353307 punch from 0x8f20a5 to 0x8f4c10 (0x2b6c bytes) 000410[1] 1790734861.363182 punch from 0x797db6 to 0x798fe8 (0x1233 bytes) 000411[1] 1790734861.406654 mapwrite 0x9bd000 thru 0x9c0fed (0x3fee bytes) 000412[1] 1790734861.491675 trunc from 0x9c0fee to 0x34e21d (0x672dd0 bytes) 000414[1] 1790734861.535665 mapread 0x1fc000 thru 0x2057f7 (0x97f8 bytes) 000415[0] 1790734861.599136 mapwrite 0x737000 thru 0x73b28a (0x428b bytes) 000416[1] 1790734861.844650 write 0x55f000 thru 0x56d01a (0xe01b bytes) 000419[0] 1790734861.928953 read 0x267000 thru 0x271e68 (0xae69 bytes) 000421[0] 1790734862.013731 mapread 0x1d5000 thru 0x1e08b9 (0xb8ba bytes) 000422[1] 1790734862.066348 mapwrite 0x53c000 thru 0x54b698 (0xf699 bytes) 000423[0] 1790734862.208851 punch from 0x05821e to 0x059662 (0x1445 bytes) 000424[1] 1790734862.241864 punch from 0x539db4 to 0x547125 (0xd372 bytes) 000425[1] 1790734862.267870 read 0x40d000 thru 0x419d48 (0xcd49 bytes) 000426[1] 1790734862.280615 read 0x2d1000 thru 0x2dc796 (0xb797 bytes) 000427[0] 1790734862.319103 read 0x0bf000 thru 0x0c18fd (0x28fe bytes) 000428[0] 1790734862.338685 mapwrite 0x7e7000 thru 0x7f33b6 (0xc3b7 bytes) 000430[0] 1790734862.453618 punch from 0x21af4f to 0x228585 (0xd637 bytes) 000432[0] 1790734862.478975 punch from 0x3e6857 to 0x3f4607 (0xddb1 bytes) 000433[0] 1790734862.483209 read 0x6b1000 thru 0x6beb14 (0xdb15 bytes) 000434[0] 1790734862.496781 read 0x792000 thru 0x797b73 (0x5b74 bytes) 000435[1] 1790734862.556560 mapread 0x11c000 thru 0x123598 (0x7599 bytes) 000438[0] 1790734862.624997 write 0x57b000 thru 0x57daf2 (0x2af3 bytes) 000439[1] 1790734862.644830 trunc from 0x7f33b7 to 0x45eb69 (0x39484d bytes) 000440[1] 1790734862.705101 mapread 0x24a000 thru 0x24e1d4 (0x41d5 bytes) 000441[0] 1790734862.745851 write 0x898000 thru 0x8a321a (0xb21b bytes) 000442[0] 1790734862.777983 punch from 0x4ee990 to 0x4fd86c (0xeedd bytes) 000443[0] 1790734862.785878 trunc from 0x8a321b to 0x0ce7e5 (0x7d4a35 bytes) 000444[1] 1790734862.826680 trunc from 0x0ce7e6 to 0x2ebe6b (0x21d686 bytes) 000445[1] 1790734862.862966 trunc from 0x2ebe6c to 0x1f7361 (0xf4b0a bytes) 000449[1] 1790734862.948820 trunc from 0x1f7362 to 0x7aa862 (0x5b3501 bytes) 000451[1] 1790734862.975313 mapwrite 0x0ed000 thru 0x0f28e0 (0x58e1 bytes) 000452[0] 1790734863.073078 read_OD 0x582000 thru 0x586fff (0x5000 bytes) 000453[0] 1790734863.101147 write_OD 0x2d8000 thru 0x2e4fff (0xd000 bytes) 000458[1] 1790734863.231231 trunc from 0x7aa863 to 0x573ac7 (0x236d9b bytes) 000459[1] 1790734863.289701 mapread 0x40b000 thru 0x412384 (0x7385 bytes) 000464[0] 1790734863.429701 write_OD 0x211000 thru 0x219fff (0x9000 bytes) 000465[0] 1790734863.616743 write_OD 0x1ed000 thru 0x1eefff (0x2000 bytes) 000466[0] 1790734863.681300 punch from 0x3d59e8 to 0x3d8b26 (0x313f bytes) 000467[0] 1790734863.699383 mapwrite 0x90b000 thru 0x90fec0 (0x4ec1 bytes) 000469[1] 1790734863.835326 read 0x64c000 thru 0x657063 (0xb064 bytes) 000476[0] 1790734863.868920 write_OD 0x798000 thru 0x798fff (0x1000 bytes) 000477[1] 1790734863.925468 mapwrite 0x0c2000 thru 0x0ca30c (0x830d bytes) 000478[0] 1790734864.051722 trunc from 0x90fec1 to 0x080a90 (0x88f430 bytes) 000479[0] 1790734864.109075 mapwrite 0x0af000 thru 0x0b93ff (0xa400 bytes) 000480[1] 1790734864.189952 mapread 0x091000 thru 0x09d5c8 (0xc5c9 bytes) 000481[1] 1790734864.240270 write 0x3b2000 thru 0x3bc114 (0xa115 bytes) 000482[1] 1790734864.270043 read 0x088000 thru 0x093506 (0xb507 bytes) 000483[0] 1790734864.309031 read_OD 0x182000 thru 0x18efff (0xd000 bytes) 000485[1] 1790734864.357261 read 0x391000 thru 0x39ed90 (0xdd91 bytes) 000487[1] 1790734864.419671 read 0x105000 thru 0x112d39 (0xdd3a bytes) 000490[1] 1790734864.420644 read 0x1cd000 thru 0x1dc54a (0xf54b bytes) 000495[1] 1790734864.422259 trunc from 0x3bc115 to 0x03fe77 (0x37c29d bytes) 000497[1] 1790734864.456877 mapwrite 0x069000 thru 0x06a885 (0x1886 bytes) 000498[0] 1790734864.514961 mapwrite 0x8a2000 thru 0x8a7a55 (0x5a56 bytes) 000499[0] 1790734864.794791 read_OD 0x211000 thru 0x216fff (0x6000 bytes) 000500[0] 1790734864.852553 punch from 0x51b86d to 0x524b23 (0x92b7 bytes) All operations completed A-OK! PASS 16a (59s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 22:21:10 (1790734870) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.364372 s, 23.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.364192 s, 23.0 MB/s Chance of close/open is 1 in 50 Seed set to 4873 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790734874.469847 write 0x27073d thru 0x271fda (0x189e bytes) 000003[1] 1790734874.509928 punch from 0x259ffb to 0x25a556 (0x55c bytes) 000004[1] 1790734874.522286 write 0x18f65e thru 0x190489 (0xe2c bytes) 000005[0] 1790734874.553116 write 0x992e73 thru 0x9948db (0x1a69 bytes) 000008[1] 1790734874.599079 mapread 0x79a821 thru 0x79b96c (0x114c bytes) 000009[1] 1790734874.677819 mapwrite 0x1649b8 thru 0x165660 (0xca9 bytes) 000011[1] 1790734874.764655 mapwrite 0x7ce935 thru 0x7cfedb (0x15a7 bytes) 000012[1] 1790734874.839405 mapwrite 0x32b965 thru 0x32ce7f (0x151b bytes) truncating to largest ever: 0x7501af 000013[0] 1790734874.918149 trunc from 0x9948dc to 0x7501ae (0x24472d bytes) 000015[1] 1790734874.969672 trunc from 0x7501af to 0x63ddcf (0x1123df bytes) 000016[1] 1790734875.062321 punch from 0x171fa5 to 0x1735ad (0x1609 bytes) 000017[0] 1790734875.116365 trunc from 0x63ddd0 to 0x689100 (0x4b331 bytes) 000018[0] 1790734875.196580 read 0x30124b thru 0x301c1f (0x9d5 bytes) 000022[1] 1790734875.280100 mapread 0x3ae207 thru 0x3afa45 (0x183f bytes) 000023[1] 1790734875.310593 mapread 0x0b9fee thru 0x0bbf69 (0x1f7c bytes) 000026[0] 1790734875.402031 mapwrite 0x91a7b0 thru 0x91bb20 (0x1371 bytes) 000028[0] 1790734875.527612 write 0x06797b thru 0x069573 (0x1bf9 bytes) 000030[0] 1790734875.556616 mapwrite 0x41f68c thru 0x41f699 (0xe bytes) 000031[0] 1790734875.588474 trunc from 0x91bb21 to 0x579f4c (0x3a1bd4 bytes) 000032[0] 1790734875.613503 mapwrite 0x13aec1 thru 0x13c8e5 (0x1a25 bytes) 000035[0] 1790734875.660880 punch from 0x567d3c to 0x569cdc (0x1fa1 bytes) 000036[1] 1790734875.664164 write 0x3376c4 thru 0x339172 (0x1aaf bytes) 000038[1] 1790734875.704364 mapread 0x39b237 thru 0x39bb19 (0x8e3 bytes) 000039[0] 1790734875.735003 mapwrite 0x805f3a thru 0x80655c (0x623 bytes) 000041[1] 1790734875.928086 trunc from 0x80655d to 0x503327 (0x303235 bytes) 000042[1] 1790734875.971175 punch from 0x4cbd64 to 0x4cdb40 (0x1ddd bytes) 000044[0] 1790734875.998985 punch from 0x3f5e26 to 0x3f6f3a (0x1115 bytes) 000046[1] 1790734876.029855 mapread 0x2627c6 thru 0x263e92 (0x16cd bytes) 000047[1] 1790734876.049707 trunc from 0x503328 to 0x68e388 (0x18b061 bytes) 000048[0] 1790734876.091295 mapwrite 0x9a40f8 thru 0x9a4bbb (0xac4 bytes) 000049[0] 1790734876.185555 punch from 0x73dd4a to 0x73f137 (0x13ee bytes) 000050[1] 1790734876.192609 mapwrite 0x788faf thru 0x78a408 (0x145a bytes) 000052[0] 1790734876.267621 punch from 0x74e006 to 0x74e4ac (0x4a7 bytes) 000053[1] 1790734876.293137 mapread 0x13923f thru 0x139531 (0x2f3 bytes) 000054[0] 1790734876.322771 read 0x304d79 thru 0x3069d9 (0x1c61 bytes) 000055[0] 1790734876.329091 read 0x71e786 thru 0x71f500 (0xd7b bytes) 000056[0] 1790734876.346331 mapread 0x3285b7 thru 0x32a475 (0x1ebf bytes) 000058[1] 1790734876.387201 mapwrite 0x91f49c thru 0x921212 (0x1d77 bytes) 000060[1] 1790734876.528990 read 0x2a25fa thru 0x2a4035 (0x1a3c bytes) 000061[1] 1790734876.564143 mapwrite 0x710352 thru 0x71154b (0x11fa bytes) 000065[0] 1790734876.678875 mapwrite 0x74e6a0 thru 0x74fc0e (0x156f bytes) 000067[0] 1790734876.780762 read 0x031cec thru 0x0324ed (0x802 bytes) 000068[1] 1790734876.800737 write 0x3156dc thru 0x317129 (0x1a4e bytes) truncating to largest ever: 0x7ab19e 000069[1] 1790734876.838471 trunc from 0x9a4bbc to 0x7ab19d (0x1f9a1e bytes) 000070[1] 1790734876.918553 mapread 0x765341 thru 0x7666ab (0x136b bytes) truncating to largest ever: 0x8aad9b 000073[1] 1790734876.948407 trunc from 0x7ab19e to 0x8aad9a (0xffbfd bytes) 000075[1] 1790734876.967115 punch from 0x0a4a41 to 0x0a5d2a (0x12ea bytes) 000077[1] 1790734876.996145 mapread 0x1c380c thru 0x1c5574 (0x1d69 bytes) 000079[0] 1790734877.020995 punch from 0x6364f6 to 0x6378d4 (0x13df bytes) 000080[1] 1790734877.049766 trunc from 0x8aad9b to 0x61097e (0x29a41c bytes) truncating to largest ever: 0x971896 000083[1] 1790734877.164722 trunc from 0x61097f to 0x971895 (0x360f17 bytes) 000084[0] 1790734877.213118 mapread 0x5dd176 thru 0x5dd22c (0xb7 bytes) 000085[1] 1790734877.250083 trunc from 0x971896 to 0x833df4 (0x13daa1 bytes) 000092[0] 1790734877.389235 write 0x6aac40 thru 0x6ab8af (0xc70 bytes) 000093[0] 1790734877.431662 write 0x4fd41a thru 0x4ff2a0 (0x1e87 bytes) 000095[0] 1790734877.474518 write 0x6857f7 thru 0x68671b (0xf25 bytes) 000097[1] 1790734877.520569 punch from 0x099c44 to 0x09b730 (0x1aed bytes) 000100[0] 1790734877.579750 punch from 0x5a64af to 0x5a7284 (0xdd6 bytes) 000101[0] 1790734877.602850 write 0x6f771f thru 0x6f78fc (0x1de bytes) 000102[1] 1790734877.625171 mapread 0x212cec thru 0x2132ca (0x5df bytes) 000103[1] 1790734877.676138 mapwrite 0x78cdd2 thru 0x78e333 (0x1562 bytes) 000105[1] 1790734877.839544 read 0x38ab00 thru 0x38aef1 (0x3f2 bytes) 000107[0] 1790734877.909648 punch from 0x6c825a to 0x6c9e80 (0x1c27 bytes) 000108[1] 1790734877.958006 write 0x875eb7 thru 0x8760c9 (0x213 bytes) 000109[1] 1790734877.982781 write 0x9ad5eb thru 0x9ae444 (0xe5a bytes) 000110[0] 1790734877.999729 read 0x2df8fa thru 0x2dfdab (0x4b2 bytes) 000116[0] 1790734878.064381 mapwrite 0x1ad515 thru 0x1aee43 (0x192f bytes) 000117[0] 1790734878.125014 mapwrite 0x9ef20d thru 0x9f0775 (0x1569 bytes) 000118[1] 1790734878.203827 punch from 0x7420c8 to 0x743445 (0x137e bytes) 000119[0] 1790734878.249709 trunc from 0x9f0776 to 0x008e73 (0x9e7902 bytes) 000120[1] 1790734878.332873 punch from 0x0055b4 to 0x005606 (0x53 bytes) 000122[1] 1790734878.375806 read 0x0078a1 thru 0x008192 (0x8f2 bytes) 000123[0] 1790734878.403026 punch from 0x002735 to 0x002ebd (0x789 bytes) 000124[1] 1790734878.420503 write 0x050491 thru 0x050c5e (0x7ce bytes) 000126[1] 1790734878.460247 write 0x9bf655 thru 0x9c0e63 (0x180f bytes) 000127[0] 1790734878.482839 trunc from 0x9c0e64 to 0x3a238b (0x61ead8 bytes) 000129[0] 1790734878.526711 mapread 0x34cceb thru 0x34d4b9 (0x7cf bytes) 000130[0] 1790734878.557345 punch from 0x243965 to 0x243aa0 (0x13c bytes) 000131[1] 1790734878.562933 punch from 0x22d5a0 to 0x22e400 (0xe61 bytes) 000133[0] 1790734878.635958 read 0x3978b0 thru 0x39974c (0x1e9d bytes) 000136[1] 1790734878.788302 punch from 0x1b13ff to 0x1b1d74 (0x976 bytes) 000137[0] 1790734878.840909 write 0x6a07ae thru 0x6a08ef (0x142 bytes) 000139[1] 1790734878.909283 punch from 0x35e33f to 0x35e98c (0x64e bytes) 000140[0] 1790734878.937794 mapwrite 0x9c8e63 thru 0x9c958b (0x729 bytes) 000141[1] 1790734879.055294 read 0x3b8bca thru 0x3b9ba9 (0xfe0 bytes) 000142[1] 1790734879.084034 punch from 0x353e35 to 0x355b10 (0x1cdc bytes) 000143[0] 1790734879.120041 mapwrite 0x13e77a thru 0x13e786 (0xd bytes) 000145[1] 1790734879.204864 mapread 0x2c2e4a thru 0x2c3fd7 (0x118e bytes) 000147[1] 1790734879.235249 mapread 0x462153 thru 0x463608 (0x14b6 bytes) 000150[1] 1790734879.264736 write 0x24ec72 thru 0x24f1d8 (0x567 bytes) 000151[1] 1790734879.290726 punch from 0x1b5172 to 0x1b69ba (0x1849 bytes) 000152[0] 1790734879.295317 mapread 0x688705 thru 0x68a1bf (0x1abb bytes) 000155[0] 1790734879.354732 mapread 0x6e1655 thru 0x6e361b (0x1fc7 bytes) 000158[1] 1790734879.389541 punch from 0x1cf101 to 0x1d0ea2 (0x1da2 bytes) 000161[0] 1790734879.413203 read 0x987bfd thru 0x9890e8 (0x14ec bytes) 000164[0] 1790734879.453625 mapread 0x96efa9 thru 0x96f037 (0x8f bytes) 000166[0] 1790734879.464692 write 0x2c06c2 thru 0x2c1230 (0xb6f bytes) 000167[0] 1790734879.504652 mapwrite 0x65b166 thru 0x65ce32 (0x1ccd bytes) 000169[1] 1790734879.581563 punch from 0x315e3e to 0x317c16 (0x1dd9 bytes) 000172[0] 1790734879.657625 mapwrite 0x72b3c9 thru 0x72c2c9 (0xf01 bytes) 000173[1] 1790734879.805137 read 0x36e62b thru 0x370444 (0x1e1a bytes) 000174[1] 1790734879.838833 trunc from 0x9c958c to 0x8370f3 (0x192498 bytes) 000175[0] 1790734879.927680 trunc from 0x8370f4 to 0x8a451f (0x6d42c bytes) 000176[0] 1790734880.000165 trunc from 0x8a4520 to 0x1ec113 (0x6b840c bytes) 000178[0] 1790734880.020802 read 0x168c6e thru 0x1698eb (0xc7e bytes) 000181[1] 1790734880.136824 mapwrite 0x5314a2 thru 0x5322d6 (0xe35 bytes) 000182[1] 1790734880.319672 trunc from 0x5322d7 to 0x19ebdf (0x3936f7 bytes) 000183[0] 1790734880.341736 read 0x159764 thru 0x15a137 (0x9d4 bytes) 000184[0] 1790734880.356461 write 0x58d84e thru 0x58e927 (0x10da bytes) 000187[0] 1790734880.440652 read 0x481874 thru 0x4823a4 (0xb31 bytes) 000188[1] 1790734880.452571 read 0x4b3c07 thru 0x4b44e4 (0x8de bytes) 000191[0] 1790734880.477558 punch from 0x001801 to 0x003263 (0x1a63 bytes) 000192[1] 1790734880.494677 trunc from 0x58e928 to 0x8c1806 (0x332edf bytes) 000193[0] 1790734880.581206 mapwrite 0x564a65 thru 0x566a25 (0x1fc1 bytes) 000194[0] 1790734880.721820 write 0x4d0822 thru 0x4d1e20 (0x15ff bytes) 000195[0] 1790734880.745775 read 0x0477ee thru 0x0488f0 (0x1103 bytes) 000196[1] 1790734880.786397 read 0x3995ad thru 0x39b05f (0x1ab3 bytes) 000197[1] 1790734880.803828 trunc from 0x8c1807 to 0x338c32 (0x588bd4 bytes) 000198[0] 1790734880.881562 mapwrite 0x8130b0 thru 0x813ec9 (0xe1a bytes) 000199[1] 1790734881.023405 punch from 0x15cc98 to 0x15d728 (0xa91 bytes) 000201[1] 1790734881.074196 read 0x3d9b63 thru 0x3dad3b (0x11d9 bytes) 000202[0] 1790734881.093739 mapwrite 0x317b30 thru 0x317ca6 (0x177 bytes) 000204[1] 1790734881.233088 write 0x63ca63 thru 0x63cb7c (0x11a bytes) 000205[0] 1790734881.303304 mapwrite 0x4bb45f thru 0x4bbcbe (0x860 bytes) 000206[0] 1790734881.493469 read 0x6cab2b thru 0x6cb747 (0xc1d bytes) 000207[0] 1790734881.551329 write 0x0ccf16 thru 0x0cd531 (0x61c bytes) 000209[0] 1790734881.624443 punch from 0x22a2ec to 0x22bbce (0x18e3 bytes) 000212[0] 1790734881.698155 punch from 0x796b92 to 0x79799c (0xe0b bytes) 000213[0] 1790734881.708319 read 0x1682d5 thru 0x1694dd (0x1209 bytes) 000216[0] 1790734881.772996 read 0x668c69 thru 0x668db9 (0x151 bytes) 000218[0] 1790734881.779368 write 0x41112e thru 0x4120a5 (0xf78 bytes) 000221[0] 1790734881.905489 trunc from 0x813eca to 0x09f355 (0x774b74 bytes) 000222[0] 1790734881.933193 trunc from 0x09f356 to 0x457357 (0x3b8002 bytes) 000226[0] 1790734882.000779 write 0x0a501f thru 0x0a6b5a (0x1b3c bytes) 000227[1] 1790734882.015301 mapwrite 0x4ecb17 thru 0x4ed62d (0xb17 bytes) 000230[1] 1790734882.156999 mapwrite 0x19b186 thru 0x19baca (0x945 bytes) 000231[0] 1790734882.228706 mapread 0x350dd4 thru 0x3526d9 (0x1906 bytes) 000234[0] 1790734882.317920 punch from 0x3923c1 to 0x394154 (0x1d94 bytes) 000235[1] 1790734882.348214 read 0x248126 thru 0x248c0e (0xae9 bytes) 000236[1] 1790734882.364605 trunc from 0x4ed62e to 0x081f45 (0x46b6e8 bytes) 000237[0] 1790734882.421245 read 0x046e22 thru 0x04784c (0xa2b bytes) 000238[0] 1790734882.474248 read 0x025bea thru 0x026ef6 (0x130d bytes) 000240[0] 1790734882.505718 read 0x07a0d6 thru 0x07ab18 (0xa43 bytes) 000245[0] 1790734882.576399 trunc from 0x081f46 to 0x7861f6 (0x7042b1 bytes) 000246[1] 1790734882.697490 mapread 0x46a19f thru 0x46a27a (0xdc bytes) 000247[1] 1790734882.755242 read 0x1502a4 thru 0x1521f0 (0x1f4d bytes) 000248[1] 1790734882.779731 mapread 0x3ff1b9 thru 0x3ff9ba (0x802 bytes) 000249[1] 1790734882.816685 punch from 0x27ab05 to 0x27c828 (0x1d24 bytes) 000250[1] 1790734882.842985 read 0x30e314 thru 0x30fb69 (0x1856 bytes) 000253[0] 1790734882.892810 mapwrite 0x580b22 thru 0x5829a9 (0x1e88 bytes) 000256[1] 1790734883.143453 trunc from 0x7861f7 to 0x244912 (0x5418e4 bytes) 000257[1] 1790734883.299801 mapread 0x0ab297 thru 0x0acda5 (0x1b0f bytes) 000258[0] 1790734883.378331 mapread 0x0afdb6 thru 0x0b13c5 (0x1610 bytes) 000260[0] 1790734883.582291 write 0x246d76 thru 0x2474dd (0x768 bytes) 000261[0] 1790734883.640790 read 0x209870 thru 0x20ad3e (0x14cf bytes) 000264[0] 1790734883.814089 mapread 0x11a03f thru 0x11a14e (0x110 bytes) 000266[0] 1790734883.886939 write 0x807087 thru 0x807e7e (0xdf8 bytes) 000267[0] 1790734883.914760 trunc from 0x807e7f to 0x4f0fbb (0x316ec3 bytes) 000269[1] 1790734884.065980 mapwrite 0x18ff55 thru 0x1906d3 (0x77f bytes) 000270[0] 1790734884.181553 mapwrite 0x783886 thru 0x785480 (0x1bfb bytes) 000273[0] 1790734884.387239 mapwrite 0x316634 thru 0x317b7e (0x154b bytes) 000276[1] 1790734884.503119 trunc from 0x785481 to 0x1c00ca (0x5c53b6 bytes) 000277[0] 1790734884.597803 read 0x02b967 thru 0x02cc0e (0x12a8 bytes) 000280[0] 1790734884.688577 punch from 0x030f1f to 0x031fec (0x10ce bytes) 000281[1] 1790734884.723275 mapwrite 0x25ae81 thru 0x25b698 (0x818 bytes) 000282[1] 1790734884.841229 punch from 0x064c2d to 0x0664ae (0x1882 bytes) 000283[0] 1790734884.873996 read 0x17fd73 thru 0x1809f0 (0xc7e bytes) 000284[0] 1790734885.000845 trunc from 0x25b699 to 0x1763b2 (0xe52e6 bytes) 000287[0] 1790734885.119815 trunc from 0x1763b3 to 0x501450 (0x38b09e bytes) 000289[1] 1790734885.232755 read 0x0355b8 thru 0x03751f (0x1f68 bytes) 000290[1] 1790734885.260453 punch from 0x019bab to 0x01b8f8 (0x1d4e bytes) 000292[1] 1790734885.347752 punch from 0x26f90b to 0x2707ee (0xee4 bytes) 000293[1] 1790734885.381874 mapwrite 0x41939a thru 0x41aff0 (0x1c57 bytes) 000294[0] 1790734885.659327 mapwrite 0x3f2030 thru 0x3f2270 (0x241 bytes) 000296[1] 1790734885.836694 mapread 0x0cea27 thru 0x0cf542 (0xb1c bytes) 000297[0] 1790734885.870991 write 0x36f214 thru 0x36f5ad (0x39a bytes) 000300[0] 1790734885.975543 punch from 0x37965f to 0x379f26 (0x8c8 bytes) 000305[0] 1790734886.115396 trunc from 0x501451 to 0x7a3c79 (0x2a2829 bytes) 000306[1] 1790734886.171188 trunc from 0x7a3c7a to 0x15b21f (0x648a5a bytes) 000307[1] 1790734886.209877 mapwrite 0x72e400 thru 0x72ec4e (0x84f bytes) 000308[1] 1790734886.274215 read 0x164ef7 thru 0x166653 (0x175d bytes) 000312[1] 1790734886.375869 trunc from 0x72ec4f to 0x7c48cd (0x95c7f bytes) 000313[1] 1790734886.400093 read 0x14af85 thru 0x14c2b9 (0x1335 bytes) 000315[0] 1790734886.406771 punch from 0x2564e7 to 0x257e3d (0x1957 bytes) 000318[0] 1790734886.475553 mapread 0x08bd0f thru 0x08d4a2 (0x1794 bytes) 000320[0] 1790734886.534383 trunc from 0x7c48ce to 0x59e320 (0x2265ad bytes) 000321[1] 1790734886.598494 write 0x95cabf thru 0x95cc27 (0x169 bytes) 000322[0] 1790734886.660764 mapread 0x6274d7 thru 0x628a89 (0x15b3 bytes) 000324[0] 1790734886.782767 read 0x304762 thru 0x306618 (0x1eb7 bytes) 000325[1] 1790734886.820756 punch from 0x507e8a to 0x509973 (0x1aea bytes) 000326[1] 1790734886.853159 read 0x065ff5 thru 0x066d36 (0xd42 bytes) 000327[1] 1790734886.877845 mapwrite 0x9010ec thru 0x902ce4 (0x1bf9 bytes) 000328[0] 1790734886.963305 mapwrite 0x132c67 thru 0x13317c (0x516 bytes) 000334[1] 1790734887.127555 punch from 0x1a4089 to 0x1a49e1 (0x959 bytes) 000336[0] 1790734887.223734 punch from 0x014767 to 0x015d86 (0x1620 bytes) 000337[1] 1790734887.265269 read 0x8ef56b thru 0x8ef8c6 (0x35c bytes) 000340[1] 1790734887.283304 mapwrite 0x6cdbcb thru 0x6ce49c (0x8d2 bytes) 000341[0] 1790734887.344735 punch from 0x0a537b to 0x0a5c40 (0x8c6 bytes) 000347[1] 1790734887.459176 punch from 0x002328 to 0x002c55 (0x92e bytes) 000348[0] 1790734887.472854 write 0x701bcb thru 0x703814 (0x1c4a bytes) 000349[0] 1790734887.505636 punch from 0x8ad3df to 0x8aed81 (0x19a3 bytes) 000350[0] 1790734887.562224 mapwrite 0x374a43 thru 0x374bdd (0x19b bytes) 000351[1] 1790734887.651315 mapwrite 0x74b002 thru 0x74baca (0xac9 bytes) 000352[0] 1790734887.762376 write 0x61cf0a thru 0x61e0b2 (0x11a9 bytes) 000353[1] 1790734887.833594 read 0x5f97d0 thru 0x5fb7a1 (0x1fd2 bytes) 000354[0] 1790734887.855372 write 0x2095e4 thru 0x20a94a (0x1367 bytes) 000355[1] 1790734887.889406 trunc from 0x95cc28 to 0x9638d6 (0x6caf bytes) 000356[1] 1790734888.008029 read 0x7954b4 thru 0x797029 (0x1b76 bytes) 000357[0] 1790734888.033380 read 0x241433 thru 0x242c79 (0x1847 bytes) 000359[0] 1790734888.085209 trunc from 0x9638d7 to 0x2d2e4a (0x690a8c bytes) 000363[1] 1790734888.229190 mapwrite 0x3ea676 thru 0x3ebd97 (0x1722 bytes) 000364[0] 1790734888.457704 punch from 0x134a6f to 0x135bf6 (0x1188 bytes) 000365[1] 1790734888.531073 write 0x64eee3 thru 0x64fc53 (0xd71 bytes) 000366[0] 1790734888.578335 write 0x1989e3 thru 0x198bc2 (0x1e0 bytes) 000371[0] 1790734888.769820 mapwrite 0x41b376 thru 0x41ce19 (0x1aa4 bytes) 000373[1] 1790734888.910203 read 0x3af15a thru 0x3af87f (0x726 bytes) 000377[0] 1790734888.967610 mapread 0x0ce233 thru 0x0cf6d3 (0x14a1 bytes) 000378[1] 1790734888.979945 mapwrite 0x40cf23 thru 0x40df0b (0xfe9 bytes) 000379[0] 1790734889.213052 mapwrite 0x69eec6 thru 0x6a0a0e (0x1b49 bytes) 000381[0] 1790734889.477718 read 0x2585c5 thru 0x25a40e (0x1e4a bytes) 000382[0] 1790734889.489450 trunc from 0x6a0a0f to 0x4d00bb (0x1d0953 bytes) 000386[1] 1790734889.572165 mapwrite 0x812e22 thru 0x814def (0x1fce bytes) 000387[1] 1790734889.761867 read 0x430e91 thru 0x43137d (0x4ed bytes) 000389[1] 1790734889.803748 mapwrite 0x2df06d thru 0x2e033c (0x12d0 bytes) 000393[1] 1790734889.916655 write 0x27c0e8 thru 0x27ccd3 (0xbec bytes) 000394[0] 1790734889.956134 trunc from 0x814df0 to 0x2e159d (0x533852 bytes) 000395[1] 1790734890.053565 punch from 0x17248f to 0x1737de (0x1350 bytes) 000396[1] 1790734890.102377 trunc from 0x2e159e to 0x3c1c42 (0xe06a5 bytes) 000398[0] 1790734890.166266 read 0x30aeba thru 0x30b575 (0x6bc bytes) 000399[1] 1790734890.272004 mapwrite 0x6a8f0e thru 0x6aade2 (0x1ed5 bytes) 000400[1] 1790734890.448666 write 0x69c80b thru 0x69cec7 (0x6bd bytes) 000401[0] 1790734890.516769 write 0x002d6c thru 0x004a81 (0x1d16 bytes) 000402[1] 1790734890.572946 write 0x21e33e thru 0x21f53e (0x1201 bytes) 000404[1] 1790734890.655270 trunc from 0x6aade3 to 0x1de1ec (0x4ccbf6 bytes) 000407[1] 1790734890.756334 punch from 0x14b7ed to 0x14becf (0x6e3 bytes) 000408[1] 1790734890.784223 read 0x0084dd thru 0x008983 (0x4a7 bytes) 000415[1] 1790734890.950612 punch from 0x1a3d5d to 0x1a557c (0x1820 bytes) 000417[0] 1790734890.998753 mapwrite 0x4ebf99 thru 0x4ed03e (0x10a6 bytes) 000420[1] 1790734891.219324 punch from 0x4baf89 to 0x4bb3b8 (0x430 bytes) 000421[1] 1790734891.293549 trunc from 0x4ed03f to 0x25761f (0x295a1f bytes) 000422[0] 1790734891.420632 read 0x086724 thru 0x0873cd (0xcaa bytes) 000423[1] 1790734891.584232 punch from 0x16cf58 to 0x16d323 (0x3cc bytes) 000424[1] 1790734891.643157 mapwrite 0x31487e thru 0x315199 (0x91c bytes) 000425[1] 1790734891.836065 mapwrite 0x7a2d7d thru 0x7a30df (0x363 bytes) 000426[0] 1790734892.020505 write 0x2b3e50 thru 0x2b48c9 (0xa7a bytes) 000427[1] 1790734892.096800 mapread 0x0b06ab thru 0x0b1906 (0x125c bytes) 000429[1] 1790734892.147837 trunc from 0x7a30e0 to 0x42f4dc (0x373c03 bytes) truncating to largest ever: 0x9f876d 000432[1] 1790734892.274202 trunc from 0x42f4dd to 0x9f876c (0x5c9290 bytes) 000434[1] 1790734892.323671 write 0x4681f0 thru 0x4682ad (0xbe bytes) 000436[0] 1790734892.350486 mapread 0x587765 thru 0x5878b8 (0x154 bytes) 000437[0] 1790734892.488321 read 0x4ebd07 thru 0x4ececb (0x11c5 bytes) 000439[0] 1790734892.506304 read 0x1f074d thru 0x1f260d (0x1ec1 bytes) 000441[0] 1790734892.526372 mapwrite 0x872bb2 thru 0x873be5 (0x1034 bytes) 000442[0] 1790734892.642506 read 0x1a6cbb thru 0x1a8298 (0x15de bytes) 000443[0] 1790734892.676712 mapwrite 0x7d7ff8 thru 0x7d9ebd (0x1ec6 bytes) 000445[1] 1790734892.815409 read 0x9ed825 thru 0x9eef17 (0x16f3 bytes) 000446[1] 1790734892.862926 punch from 0x8ac7d2 to 0x8ade78 (0x16a7 bytes) 000448[1] 1790734892.930706 read 0x052dc3 thru 0x053c60 (0xe9e bytes) 000450[0] 1790734892.966197 punch from 0x1c15d2 to 0x1c2eb4 (0x18e3 bytes) 000451[1] 1790734892.985213 punch from 0x2b2396 to 0x2b28ab (0x516 bytes) 000452[1] 1790734893.013525 trunc from 0x9f876d to 0x5c9bdb (0x42eb91 bytes) 000453[0] 1790734893.055716 mapwrite 0x975903 thru 0x975ba1 (0x29f bytes) 000454[0] 1790734893.198687 read 0x629232 thru 0x6297ab (0x57a bytes) 000456[1] 1790734893.253941 mapwrite 0x4e40e0 thru 0x4e435b (0x27c bytes) 000457[1] 1790734893.355274 trunc from 0x975ba2 to 0x838bff (0x13cfa2 bytes) 000459[1] 1790734893.440948 mapread 0x6a5a1d thru 0x6a70fa (0x16de bytes) 000460[1] 1790734893.467584 trunc from 0x838c00 to 0x527688 (0x311577 bytes) 000461[1] 1790734893.525969 trunc from 0x527689 to 0x840d9a (0x319712 bytes) 000462[0] 1790734893.556591 mapwrite 0x2efded thru 0x2f12a0 (0x14b4 bytes) 000463[0] 1790734893.689619 mapwrite 0x1ad131 thru 0x1aedef (0x1cbf bytes) 000464[0] 1790734893.853137 mapwrite 0x416bf7 thru 0x417034 (0x43e bytes) 000465[1] 1790734893.965324 trunc from 0x840d9b to 0x84ef95 (0xe1fb bytes) 000466[0] 1790734894.070252 write 0x8c2b9c thru 0x8c3e1d (0x1282 bytes) 000467[0] 1790734894.132005 mapwrite 0x35a50d thru 0x35b5a5 (0x1099 bytes) 000468[0] 1790734894.293249 trunc from 0x8c3e1e to 0x714741 (0x1af6dc bytes) 000469[0] 1790734894.345604 read 0x60aad9 thru 0x60b7cc (0xcf4 bytes) 000470[1] 1790734894.418467 mapwrite 0x6879a3 thru 0x688579 (0xbd7 bytes) 000472[1] 1790734894.547808 punch from 0x686df9 to 0x6888b8 (0x1ac0 bytes) 000473[0] 1790734894.568108 trunc from 0x714742 to 0x4c1739 (0x253008 bytes) 000475[1] 1790734894.629393 punch from 0x46b861 to 0x46c5d3 (0xd73 bytes) 000477[1] 1790734894.677064 mapread 0x23cae8 thru 0x23d45e (0x977 bytes) 000479[1] 1790734894.756486 mapwrite 0x8d27b1 thru 0x8d446f (0x1cbf bytes) 000480[0] 1790734894.879497 write 0x364cf4 thru 0x3661b7 (0x14c4 bytes) 000481[0] 1790734894.927047 read 0x53c825 thru 0x53d965 (0x1141 bytes) 000483[1] 1790734894.972291 write 0x91cb00 thru 0x91e07c (0x157d bytes) 000486[1] 1790734895.049708 read 0x5f0c89 thru 0x5f0df7 (0x16f bytes) 000487[0] 1790734895.055231 punch from 0x15562e to 0x155f75 (0x948 bytes) 000488[0] 1790734895.103062 write 0x5fe2a1 thru 0x6000ab (0x1e0b bytes) 000489[1] 1790734895.168770 punch from 0x8b23dc to 0x8b371b (0x1340 bytes) 000491[1] 1790734895.236990 mapread 0x102050 thru 0x102351 (0x302 bytes) 000492[0] 1790734895.302548 mapread 0x4abf54 thru 0x4acc90 (0xd3d bytes) 000493[1] 1790734895.368320 mapwrite 0x9ac220 thru 0x9ac95d (0x73e bytes) 000494[0] 1790734895.722335 read 0x857e08 thru 0x859b0b (0x1d04 bytes) 000495[1] 1790734895.769180 mapread 0x678beb thru 0x6799a7 (0xdbd bytes) 000496[1] 1790734895.815537 mapread 0x09bd43 thru 0x09c1b3 (0x471 bytes) 000498[0] 1790734895.861157 write 0x90eee6 thru 0x90f9d0 (0xaeb bytes) 000499[0] 1790734895.907016 trunc from 0x9ac95e to 0x712e8c (0x299ad1 bytes) 000500[1] 1790734895.943235 trunc from 0x712e8d to 0x7012bb (0x11bd1 bytes) All operations completed A-OK! PASS 16b (32s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 22:21:42 (1790734902) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 22:21:46 (1790734906) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.593968 s, 14.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.386897 s, 21.7 MB/s ldlm.namespaces.MGC192.168.201.121@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8acb8573d800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8acb910e1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8acb8573d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8acb8573d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.819 s, 7.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.58375 s, 22.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.74476 s, 22.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.8676 s, 11.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.40594 s, 11.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.28653 s, 31.9 MB/s PASS 16d (61s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 22:22:47 (1790734967) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300744 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10707 s, 9.8 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 22:22:54 (1790734974) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 22:23:23 (1790735003) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 22:23:53 (1790735033) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 22:24:02 (1790735042) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 22:24:10 (1790735050) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52367 s, 17.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.69312 s, 15.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49405 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.94256 s, 13.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72987 s, 9.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.34335 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.2281 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.28003 s, 20.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74149 s, 15.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.95451 s, 6.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.14505 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.60588 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.90679 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.26997 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.46342 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.83542 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27997 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24406 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50231 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (43s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 22:24:53 (1790735093) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Tue Sep 29 22:25:49 EDT 2026 + fsx 2 done: rc=0 Tue Sep 29 22:25:49 EDT 2026 + fsx 3 done: rc=0 All operations completed A-OK! Tue Sep 29 22:25:49 EDT 2026 + fsx 4 done: rc=0 Tue Sep 29 22:25:49 EDT 2026 + fsx 5 done: rc=0 Tue Sep 29 22:25:49 EDT 2026 PASS 16k (64s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 22:25:57 (1790735157) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 22:26:08 (1790735168) mmap test1: basic mmap operation (PASS, 0.14736s) mmap test2: MAP_PRIVATE not write back (PASS, 0.072915s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.801s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 107.69s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.22257s) mmap test6: check mmap write/read content on two nodes (PASS, 0.31233s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4219s) mmap test11: multi-thread shared mmap (PASS, 1.5849s) mmap test10: mtime not change for readonly mmap access (PASS, 1.2s) PASS 18 (136s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:28:24 (1790735304) oleg121-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:28:29 (1790735309) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:28:38 (1790735318) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 22:28:46 (1790735326) now is 1790735329 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 new atime is 1790735390 PASS 23 (74s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:30:00 (1790735400) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:30:09 (1790735409) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:30:17 (1790735417) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:30:25 (1790735425) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 22:30:28 (1790735428) utime: good mknod times 1790735428 <= 1790735429 <= 1790735429 for /mnt/lustre/f26a utime: good mknod times 1790735428 <= 1790735429 <= 1790735429 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 22:30:35 (1790735435) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:30:44 (1790735444) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-29 22:30:46.000000000 -0400 Birth: 2026-09-29 22:30:45.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-29 22:30:46.000000000 -0400 Birth: 2026-09-29 22:30:45.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:30:53 (1790735453) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:31:02 (1790735462) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:31:09 (1790735469) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg121-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.121@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 22:31:32 (1790735492) -rw-r--r-- 1 root root 0 Sep 29 22:31 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:31:44 (1790735504) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 43213 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:31:52 (1790735512) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 22:32:00 (1790735520) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 22:32:03 (1790735523) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29393 s, 40.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.28002 s, 41.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.085 s, 25.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.33347 s, 22.5 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.65856 s, 19.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.74848 s, 19.1 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57863 s, 33.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.68734 s, 31.1 MB/s PASS 32c (31s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:32:36 (1790735556) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:32:40 (1790735560) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:32:44 (1790735564) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:32:48 (1790735568) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 22:32:51 (1790735571) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid in FULL state after 0 sec oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid in IDLE state after 0 sec oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid in FULL state after 0 sec oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid in IDLE state after 0 sec oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acb8573d800.ost_server_uuid in FULL state after 0 sec oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acb8573d800.ost_server_uuid in IDLE state after 0 sec PASS 34 (143s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:35:14 (1790735714) fail_loc=0x80000317 Race attempt 0 Wait for 50517 50534 for 60 sec... - open/close 543 (time 1790735733.65 total 10.01 last 54.25) - open/close 1121 (time 1790735743.66 total 20.02 last 57.75) - open/close 1814 (time 1790735753.66 total 30.03 last 69.24) - open/close 2480 (time 1790735763.68 total 40.04 last 66.49) - open/close 3265 (time 1790735773.68 total 50.04 last 78.48) total: 4000 open/close in 58.54 seconds: 68.34 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1861: 50517 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.121@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.201.121@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:36:29 (1790735789) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg121-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27423 s, 41.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06542 s, 25.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18359 s, 44.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7445504) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.68297 s, 31.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65025 s, 31.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.89972 s, 27.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.06679 s, 49.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07046 s, 49.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.954 s, 26.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48044 s, 35.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.970431 s, 54.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7441408) after(7497728) PASS 36 (308s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:41:38 (1790736098) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 - create 1148 (time 1790736112.20 total 10.00 last 114.79) - create 2790 (time 1790736122.21 total 20.01 last 164.10) - create 4363 (time 1790736132.21 total 30.01 last 157.26) - create 5756 (time 1790736142.22 total 40.02 last 139.14) - create 7211 (time 1790736152.23 total 50.03 last 145.43) - create 8885 (time 1790736162.23 total 60.03 last 167.33) total: 10000 create in 68.07 seconds: 146.91 ops/second PASS 37 (80s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:42:57 (1790736177) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:43:05 (1790736185) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:43:14 (1790736194) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 22:43:22 (1790736202) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:43:31 (1790736211) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:43:50 (1790736230) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:44:07 (1790736247) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:44:23 (1790736263) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:44:40 (1790736280) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:44:54 (1790736294) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:45:08 (1790736308) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:45:21 (1790736321) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:45:33 (1790736333) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:45:47 (1790736347) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:46:01 (1790736361) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:46:14 (1790736374) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 22:46:26 (1790736386) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1087s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 23:04:34 (1790737474) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 23:04:46 (1790737486) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 23:04:58 (1790737498) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 23:05:11 (1790737511) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 23:05:24 (1790737524) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 23:05:37 (1790737537) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 23:05:49 (1790737549) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 23:06:01 (1790737561) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 23:06:15 (1790737575) PASS 43a (59s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 23:07:14 (1790737634) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 23:07:27 (1790737647) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 23:07:39 (1790737659) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 23:07:52 (1790737672) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 23:08:07 (1790737687) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (17s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 23:08:24 (1790737704) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 23:08:36 (1790737716) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 23:08:49 (1790737729) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 23:08:52 (1790737732) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (121s) == sanityn test 43k: unlink vs create ==================== 23:10:53 (1790737853) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1126s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 23:29:39 (1790738979) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 29 23:29:46 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 23:29:53 (1790738993) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 23:30:08 (1790739008) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 23:30:23 (1790739023) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 23:30:36 (1790739036) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 23:30:49 (1790739049) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 23:31:02 (1790739062) Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 23:31:18 (1790739078) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 23:31:31 (1790739091) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 23:31:34 (1790739094) PASS 45a (111s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 23:33:25 (1790739205) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 23:33:39 (1790739219) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 23:33:53 (1790739233) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 23:34:07 (1790739247) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 23:34:21 (1790739261) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (12s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 23:34:33 (1790739273) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 23:34:46 (1790739286) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 23:34:57 (1790739297) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 23:34:59 (1790739299) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1171s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:54:30 (1790740470) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:54:43 (1790740483) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:54:56 (1790740496) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:55:08 (1790740508) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:55:20 (1790740520) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:55:32 (1790740532) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:55:44 (1790740544) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:55:57 (1790740557) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:56:10 (1790740570) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:56:13 (1790740573) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:56:16 (1790740576) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:56:20 (1790740580) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:56:23 (1790740583) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:56:26 (1790740586) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:56:30 (1790740590) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:56:33 (1790740593) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:56:36 (1790740596) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.166158 s, 61.6 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:56:49 (1790740609) 0+1 records in 0+1 records out 158 bytes copied, 0.0275572 s, 5.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (218501) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:56:58 (1790740618) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00945213 s, 108 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656125 s, 16.0 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:57:18 (1790740638) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0245031 s, 41.8 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:57:31 (1790740651) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593922 s, 17.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:57:39 (1790740659) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2449 0x991 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2449 0x991 0x280000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 23:57:48 (1790740668) llite.lustre-ffff8acb8573d800.xattr_cache=1 llite.lustre-ffff8acb910e1000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:57:50.000000000 -0400 Modify: 2026-09-29 23:57:50.000000000 -0400 Change: 2026-09-29 23:57:50.000000000 -0400 Birth: 2026-09-29 23:57:50.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:57:57.000000000 -0400 Modify: 2026-09-29 23:57:58.000000000 -0400 Change: 2026-09-29 23:57:58.000000000 -0400 Birth: 2026-09-29 23:57:57.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:58:03.000000000 -0400 Modify: 2026-09-29 23:58:03.000000000 -0400 Change: 2026-09-29 23:58:03.000000000 -0400 Birth: 2026-09-29 23:58:03.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:58:03.000000000 -0400 Modify: 2026-09-29 23:58:04.000000000 -0400 Change: 2026-09-29 23:58:04.000000000 -0400 Birth: 2026-09-29 23:58:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:58:10.000000000 -0400 Modify: 2026-09-29 23:58:11.000000000 -0400 Change: 2026-09-29 23:58:11.000000000 -0400 Birth: 2026-09-29 23:58:10.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:58:10.000000000 -0400 Modify: 2026-09-29 23:58:11.000000000 -0400 Change: 2026-09-29 23:58:11.000000000 -0400 Birth: 2026-09-29 23:58:10.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 23:58:22 (1790740702) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 23:58:36 (1790740716) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 23:58:50 (1790740730) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 23:59:10 (1790740750) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:59:26 (1790740766) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:59:30 (1790740770) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (22s) == sanityn test 55g: rename: race with trylock =========== 23:59:52 (1790740792) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:00:16 (1790740816) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790740819 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.586914 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (113s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 00:02:11 (1790740931) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 00:02:20 (1790740940) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:02:28 (1790740948) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:02:35 (1790740955) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00706392 s, 9.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:02:38 (1790740958) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0102933 s, 6.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:02:41 (1790740961) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:02:44 (1790740964) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0368822 s, 1.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:02:48 (1790740968) llite.lustre-ffff8acb8573d800.xattr_cache=1 llite.lustre-ffff8acb910e1000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:02:55 (1790740975) llite.lustre-ffff8acb8573d800.xattr_cache=1 llite.lustre-ffff8acb910e1000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:03:02 (1790740982) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0125742 s, 81.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0264621 s, 38.7 kB/s 230878: get lock2 230879: get lock1 230878: done 230878: get lock1 230879: done 230879: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 230879: cannot get lock2: Resource deadlock avoided 230879: exit rc=0 230878: done 230878: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:03:14 (1790740994) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118483 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609916 s, 17.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 231461 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 00:03:27 (1790741007) opening files: ........ closing files: ........ PASS 76 (68s) == sanityn test 77a: check FIFO NRS policy =============== 00:04:36 (1790741076) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496506 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101876 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339436 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125498 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091434 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139407 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890184 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194568 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11397 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137955 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210379 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231671 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280035 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338557 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265136 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38878 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355334 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120059 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115757 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393682 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953713 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112499 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438887 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10976 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122614 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596575 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147772 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114881 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740442 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759018 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120973 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161035 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139882 s, 7.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 00:04:45 (1790741085) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.448473 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792427 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845964 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188043 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215117 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264287 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309041 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26859 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215481 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316528 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356075 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382609 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325829 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356808 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403313 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373961 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474402 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237421 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250366 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00259995 s, 403 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254309 s, 412 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176493 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024856 s, 422 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265923 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150046 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123519 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145325 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124004 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975426 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119063 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107436 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105408 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094415 s, 11.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567943 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110418 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124687 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137786 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174254 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225207 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189993 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22062 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137596 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210991 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288871 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100068 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318243 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386188 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433412 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315614 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403798 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554466 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14148 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117603 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00221662 s, 473 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810718 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672604 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149383 s, 70.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486165 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749049 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487937 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06101 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450964 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401952 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042266 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339577 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032968 s, 31.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 00:04:58 (1790741098) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.488522 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589601 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107518 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115404 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117917 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155968 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224603 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245942 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278945 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262696 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25973 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337863 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339205 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32813 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411569 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484064 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438024 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150757 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984962 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545136 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214636 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305212 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657462 s, 15.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103469 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448322 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255364 s, 411 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105663 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152151 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398074 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805912 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115945 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243923 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236774 s, 4.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618177 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733569 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663117 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133096 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834661 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114439 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251459 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153028 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799108 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175239 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439585 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256544 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313687 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258021 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162626 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286366 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222617 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297447 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867776 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374862 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352394 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422011 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754227 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459776 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490989 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466865 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836918 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534591 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768784 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579289 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618343 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664444 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691168 s, 15.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44783 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889565 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754299 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115718 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178449 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184713 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145671 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197013 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140141 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700511 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235361 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397354 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25252 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249728 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392101 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425151 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345881 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112482 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00981376 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100411 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674678 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601295 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421485 s, 249 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727895 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180321 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635094 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037695 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103924 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618221 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363978 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314987 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139564 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204356 s, 51.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 00:05:16 (1790741116) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.398802 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107613 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142264 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154936 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267008 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250724 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227323 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305212 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409356 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351356 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40427 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38472 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510791 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505746 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499649 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39068 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166331 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149589 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930882 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174974 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932937 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165233 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167135 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111364 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155677 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106152 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106685 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118357 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165339 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141663 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.640933 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189165 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203072 s, 5.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426405 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116444 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155575 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22281 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233354 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255936 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420019 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297549 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329178 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297821 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423089 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493572 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433805 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497775 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53303 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.519223 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503533 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00263427 s, 398 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0023542 s, 445 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248844 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568633 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431602 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222602 s, 471 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627303 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471853 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189928 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114868 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204519 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00694563 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260493 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319445 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399159 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623739 s, 16.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.438907 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892961 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105052 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207921 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176656 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152287 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259147 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195735 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342922 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324337 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298867 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294304 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341415 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400758 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367513 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403314 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459167 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489569 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00260842 s, 402 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239901 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688984 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209076 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246261 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145483 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122646 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119244 s, 87.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285369 s, 367 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130261 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112496 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100327 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207115 s, 50.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257053 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062104 s, 16.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 00:05:32 (1790741132) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.21@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428025 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657235 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718598 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310994 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973539 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12705 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553136 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127259 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185593 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238419 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160827 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110395 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244193 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215841 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2024 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175735 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446754 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00309537 s, 339 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755862 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100097 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642848 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404411 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182122 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912192 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768274 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974138 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115367 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175645 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273725 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947337 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183767 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140818 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563243 s, 18.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499867 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884685 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636187 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10269 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16862 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135525 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217296 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112833 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548639 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156343 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138092 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113023 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266691 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498632 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592486 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069234 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160736 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761071 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669288 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789089 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062177 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718094 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901044 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1035 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064033 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774721 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593143 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825899 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875699 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104966 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717989 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979237 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912377 s, 11.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.509019 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616561 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858781 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932431 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231846 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230869 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298681 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242097 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237066 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271354 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323952 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312551 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305218 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330613 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336468 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35976 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380627 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965901 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075089 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441789 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441259 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480192 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439942 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529369 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519596 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524252 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686162 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564186 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664374 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668514 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658557 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800563 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806657 s, 13.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585846 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838136 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113489 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256709 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289603 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528526 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355142 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234385 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185214 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418352 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186312 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436187 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456581 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244402 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454093 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205012 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189729 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131374 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15265 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431885 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160309 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293942 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486774 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861497 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496951 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515771 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523758 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285265 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350993 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133897 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479297 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405016 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529389 s, 2.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (23s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:05:56 (1790741156) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402378 s, 41.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374433 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947938 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292504 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105511 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179652 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195413 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130429 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264317 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194716 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281604 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275292 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227357 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235029 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317513 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283445 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102141 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08135 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752604 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244692 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106814 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406134 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052687 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098061 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112986 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10434 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799098 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703672 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853017 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894291 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910443 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165558 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109664 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424012 s, 39.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122804 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137387 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130398 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216708 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256746 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162768 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230535 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203619 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319582 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271961 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279041 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275326 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245594 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365236 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365287 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378143 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500395 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234671 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545937 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277738 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312559 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289161 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030544 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324345 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319372 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044786 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311394 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409769 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364845 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349722 s, 30.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046296 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422353 s, 24.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501759 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139237 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145769 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185464 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2529 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233351 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321269 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335599 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320408 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417424 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191681 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472661 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536054 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511339 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.606845 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.615558 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.622137 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274889 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731571 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541391 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771824 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357663 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361064 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168583 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609488 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419337 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105558 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121773 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157393 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108718 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183184 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976837 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117607 s, 8.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450924 s, 37.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491618 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476747 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770184 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155585 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602029 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135138 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221192 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260627 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162765 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154088 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273905 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214866 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198472 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184436 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793762 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217302 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101458 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618599 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057661 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678745 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594727 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623835 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823608 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072183 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768107 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611517 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828887 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580707 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091067 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141019 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984733 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080928 s, 13.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (25s) == sanityn test 77g: Change TBF type directly ============ 00:06:20 (1790741180) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (10s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:06:31 (1790741191) oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 00:06:44 (1790741204) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:07:05 (1790741225) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.495727 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850468 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977713 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526531 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16804 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179204 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153164 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208723 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205352 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239803 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275214 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20867 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250588 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229825 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261929 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260502 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255029 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258698 s, 405 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00260285 s, 403 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154138 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127074 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641127 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109561 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133456 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955182 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621171 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753864 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053707 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168676 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519392 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432697 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499822 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339483 s, 30.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.79319 s, 15.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6238 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (67s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:08:12 (1790741292) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31912 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089295 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132935 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795572 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169432 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190778 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232222 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204596 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301951 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273175 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350803 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337864 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37737 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418974 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410531 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409349 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463881 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591329 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688367 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069519 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104949 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104191 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764456 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088092 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116546 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056967 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132559 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886891 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703953 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103081 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811493 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866454 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104982 s, 10.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9263 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9869 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28566 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205104 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984794 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127699 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104584 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151357 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169162 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180745 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189867 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118179 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254713 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16294 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263613 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246932 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161173 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277424 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194142 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955491 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877422 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117515 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15994 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16623 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102797 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178092 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178448 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145806 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139521 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163374 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204254 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158846 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216603 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156389 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188147 s, 5.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0158 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9586 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (157s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:10:49 (1790741449) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3063 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109931 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178361 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161569 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209985 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244235 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189512 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223444 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356587 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383854 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438724 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488551 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461019 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531765 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.569763 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.537128 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.564429 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922615 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102227 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949528 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116737 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918646 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122954 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120633 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126903 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11739 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114869 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16155 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15536 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129945 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144638 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156704 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170011 s, 6.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9364 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9766 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29104 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100214 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122731 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182055 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153608 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20719 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144589 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088405 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063753 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191495 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136232 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372282 s, 2.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26796 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336404 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376396 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235066 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369212 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872697 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642562 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653286 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996774 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761171 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770043 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913374 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056244 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844927 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105863 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892994 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126846 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127984 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801303 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134184 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101056 s, 10.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0416 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9591 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (152s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:13:21 (1790741601) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463573 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709753 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959511 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155333 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136087 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191497 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176513 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132769 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119258 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15962 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132654 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162358 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166296 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150361 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148125 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212314 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181664 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024316 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119639 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586135 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103949 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674125 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882179 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991852 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789399 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380591 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131461 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739815 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524712 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580449 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593176 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602951 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108826 s, 9.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6147 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91449 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.21@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.21@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522688 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130586 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151098 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16901 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189102 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237164 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317714 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328084 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388319 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360898 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37608 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446388 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466465 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4126 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444983 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522444 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53982 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181711 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156389 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167845 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22592 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236177 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160823 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247634 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205338 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183153 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172418 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189638 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235118 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274122 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205641 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221034 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200957 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=4 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.11125 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93881 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.21@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464061 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769425 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878921 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496459 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17257 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179247 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156546 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222075 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185733 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260735 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267674 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278094 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246759 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326197 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296153 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305837 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269406 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399033 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749363 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258132 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030083 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427928 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503974 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317047 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413642 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421748 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428687 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448315 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475754 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632087 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460881 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580876 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904064 s, 11.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.81248 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27461 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.740783 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861278 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747334 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740688 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177342 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147455 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220624 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136038 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211422 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23319 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203524 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225278 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206197 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270934 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231965 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269716 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296615 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373362 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396526 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408561 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685378 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585222 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402565 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491621 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651462 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483233 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525266 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651835 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866027 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628497 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870602 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619824 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694708 s, 15.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7484 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91515 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39322 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98312 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03536 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669494 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573189 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920025 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250391 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271907 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297524 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313085 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315283 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31035 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249219 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277058 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316447 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314779 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307249 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296256 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382979 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667431 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805298 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121839 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771508 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988983 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672389 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113403 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108303 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793185 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107518 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20416 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894463 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11254 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07847 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11501 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122306 s, 8.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8228 s, 5.3 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.614 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501715 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104729 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609329 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138208 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115392 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139725 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175617 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222216 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226694 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243379 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249863 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237436 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199104 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207218 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288638 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308348 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354496 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739325 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758853 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103525 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863356 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315528 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370491 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695914 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395304 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404966 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424394 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460841 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487266 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607977 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059181 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662059 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676515 s, 15.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35454 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96246 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.891695 s, 18.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628414 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899773 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125504 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887217 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188505 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197431 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230529 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234386 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242792 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231343 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227127 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272238 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209989 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199848 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280478 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246952 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535695 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386612 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239918 s, 437 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366473 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00597997 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914075 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904393 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676115 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059285 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781911 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110991 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132988 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343032 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034736 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111419 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377043 s, 27.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6183 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90553 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.851 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9206 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (434s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:20:36 (1790742036) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.395336 s, 42.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788769 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10057 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21124 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190113 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265796 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249866 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328522 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308428 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215771 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313057 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284623 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379653 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30911 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4226 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39236 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336783 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037496 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371223 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405099 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214545 s, 48.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317657 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340724 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349907 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311995 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350649 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376716 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404848 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573814 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395479 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442505 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602413 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516133 s, 20.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0044 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90955 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:21:28 (1790742088) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.21@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.21@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.624093 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799518 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140577 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146689 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152738 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141521 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150137 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244985 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331409 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354684 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237372 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384887 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360891 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36027 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477264 s, 2.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462122 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567457 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198582 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149174 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23328 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155068 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158777 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185644 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20048 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232746 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27092 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199724 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202057 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201898 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275351 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21383 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224169 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231951 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.10341 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94802 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (54s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:22:22 (1790742142) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.21@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06904 s, 15.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075297 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759056 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209729 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248559 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274555 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb910e1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306109 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313163 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342685 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354518 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361503 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366716 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388589 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376584 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408563 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421814 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455936 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883201 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123979 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043432 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437718 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015131 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707165 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107734 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902954 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121947 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111688 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471865 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682611 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124112 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826886 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283126 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175028 s, 59.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.8675 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.50342 s, 19.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:23:09 (1790742189) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.657638 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849661 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980626 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858722 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152553 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145796 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252277 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322619 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320239 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262818 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290421 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311682 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258703 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25704 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29487 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250092 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357015 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659131 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696225 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728498 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660546 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764271 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070898 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062247 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810468 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916479 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129032 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110948 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120078 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104816 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107928 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171206 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146746 s, 7.1 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6498 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95983 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48568 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96883 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (94s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:24:43 (1790742283) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08464 s, 15.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892549 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08922 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12529 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051908 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167628 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874321 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176586 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102126 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23501 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649241 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676367 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227275 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363078 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365086 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321476 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355735 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105283 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123044 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167231 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110405 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156243 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176606 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126537 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115561 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130038 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168939 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129895 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107779 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138016 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209076 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147271 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240613 s, 4.4 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8668 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9928 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (75s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:25:59 (1790742359) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.714173 s, 23.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808489 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127053 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133823 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182893 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179669 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126848 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163919 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118293 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339145 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265491 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342728 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272577 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407649 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345254 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430805 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb910e1000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601465 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490976 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052369 s, 200 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875278 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780772 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766435 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595069 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13893 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583827 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525633 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107555 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728985 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11051 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927393 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106752 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095873 s, 10.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.34013 s, 12.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0222 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg121-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.62265 s, 26.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073565 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901121 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131832 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166252 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134601 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161785 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103663 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111076 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188446 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233825 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272783 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293674 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248921 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326609 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416195 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447761 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626447 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649087 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00762404 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943317 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952302 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343751 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493831 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375873 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423515 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034487 s, 304 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849209 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02073 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033456 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522938 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577249 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183389 s, 57.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8936 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0807 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.617 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1488 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (144s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 00:28:22 (1790742502) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:) Stopping client oleg121-client.virtnet /mnt/lustre opts: Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg121-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg121-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg121-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.21,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg121-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.21,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg121-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.21,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71107/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.21,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg121-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: kmemleak disabled oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre Starting client oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre Started clients oleg121-client.virtnet: 192.168.201.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Starting client oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Started clients oleg121-client.virtnet: 192.168.201.121@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8acb843b7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8acb9097b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb843b7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb9097b000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409762 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477088 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784372 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106284 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564202 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303758 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163646 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270936 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471554 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170993 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369614 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124213 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243764 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166653 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179784 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147725 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176565 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663309 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192608 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864257 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593721 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916879 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100631 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436218 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606747 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623194 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329877 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237333 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552312 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637591 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795993 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471227 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718919 s, 14.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.44628 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23719 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.596612 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158742 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141217 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371317 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343292 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442167 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374037 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347316 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323359 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409972 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368789 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39218 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354662 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412305 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484627 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358616 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422354 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20995 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210338 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21398 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227128 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226405 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229129 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2236 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238982 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242606 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252597 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235051 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239985 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236177 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263202 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258124 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272895 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.87012 s, 17.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92504 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (219s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 00:32:01 (1790742721) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 00:32:18 (1790742738) On MGS 192.168.201.121, active = nodemap.active=1 On MGS 192.168.201.121, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.51703 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422125 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853121 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346258 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109383 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944645 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540426 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2975 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300787 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290708 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174028 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289377 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325455 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379936 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289032 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296316 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314142 s, 3.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325469 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326193 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307185 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314626 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323264 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317424 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318394 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329485 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321568 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33331 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338293 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333491 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389712 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347189 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344516 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369603 s, 2.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.61021 s, 15.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41104 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.121, TBF.id = On MGS 192.168.201.121, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.808353 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118552 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194755 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179526 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254938 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514982 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837802 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665938 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342116 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169258 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44627 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231993 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373778 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399321 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468047 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408508 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512409 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398725 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240611 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322903 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345231 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690638 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388352 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108136 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012271 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455905 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727908 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990404 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594814 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104011 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632094 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651799 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132859 s, 7.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8314 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.84989 s, 36.8 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg121-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 On MGS 192.168.201.121, active = nodemap.active=1 On MGS 192.168.201.121, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.668471 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044168 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101864 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17805 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196695 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164487 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160473 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323064 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317368 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345936 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327934 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331364 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412643 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301163 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335051 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431511 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412191 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889266 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172609 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101588 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100068 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119443 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125405 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144691 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120255 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148517 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156238 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132188 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110545 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121316 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152252 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135736 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176191 s, 6.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7609 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0287 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.121, TBF.id = On MGS 192.168.201.121, active = nodemap.active=0 PASS 77kj (185s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:35:23 (1790742923) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:35:33 (1790742933) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg121-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.45465 s, 304 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg121-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0985 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:36:31 (1790742991) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.471405 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108632 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102917 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748084 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238469 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239043 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285182 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221685 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308216 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365426 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34011 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378062 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376584 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472887 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493505 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455246 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402189 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343813 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737268 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217504 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516129 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597863 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672578 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712918 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072229 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07267 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788587 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830698 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834463 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866995 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899083 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968595 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104968 s, 10.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg121-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg121-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.53273 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28689 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537798 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950795 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115136 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181669 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229921 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379971 s, 2.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402135 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398872 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351023 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393616 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451161 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443731 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444274 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492933 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489736 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544252 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480841 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8acb843b7800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8acb843b7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146177 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124008 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694193 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716824 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157792 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577894 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824654 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156712 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951292 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923558 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19591 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104037 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103021 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104353 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900177 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971568 s, 10.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg121-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg121-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.45986 s, 16.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24175 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (87s) == sanityn test 77o: Changing rank should not panic ====== 00:37:58 (1790743078) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:38:10 (1790743090) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (119s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:40:09 (1790743209) mds.MDS.mdt.nrs_policies=tbf oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg121-client: oleg121-server: ssh exited with exit code 36 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 oleg121-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg121-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:40:44 (1790743244) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272442 kill md_thread xargs PASS 77r (54s) == sanityn test 77s: Check TBF LRU shrinker ============== 00:41:38 (1790743298) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (56s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:42:34 (1790743354) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 00:42:44 (1790743364) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 2 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 00:42:47.000000000 -0400 Modify: 2026-09-30 00:42:47.000000000 -0400 Change: 2026-09-30 00:42:47.000000000 -0400 Birth: 2026-09-30 00:42:46.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:43:03 (1790743383) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:43:06 (1790743386) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:43:09 (1790743389) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:43:12 (1790743392) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:43:15 (1790743395) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:43:18 (1790743398) - open/close 908 (time 1790743414.32 total 10.01 last 90.75) - open/close 1874 (time 1790743424.33 total 20.01 last 96.54) total: 2500 open/close in 27.38 seconds: 91.29 ops/second snapshot_time 1790743518.866593719 secs.nsecs start_time 1790743401.529183402 secs.nsecs elapsed_time 117.337410317 secs.nsecs open 2505 samples [usecs] 114 17278 3170853 6197010579 close 2504 samples [usecs] 153 9194 791188 401108136 mknod 2501 samples [usecs] 520 16863 2791652 4995976454 unlink 2500 samples [usecs] 549 84849 5621859 66769317797 mkdir 2 samples [usecs] 1020 2210 3230 5924500 rmdir 2 samples [usecs] 7009 22567 29576 558395570 rename 2500 samples [usecs] 375 69786 16582102 243912744094 getattr 3762 samples [usecs] 6 18790 269907 939073395 parallel_rename_file 2500 samples [usecs] 375 69786 16582102 243912744094 crossdir_rename 2500 samples [usecs] 375 69786 16582102 243912744094 readpage 26 samples [usecs] 4060 67533 593076 20156629558 PASS 81d (127s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:45:25 (1790743525) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7624 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7624 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:45:32 (1790743532) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:45:35 (1790743535) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 00:45:49 (1790743549) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790743551 Finishing test test1 at 1790743555 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:46:01 (1790743561) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 00:46:04 (1790743564) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:46:08 (1790743568) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:46:11 (1790743571) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4796 0x12bc 0x240000400 1 4759 0x1297 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4797 0x12bd 0x240000400 1 4760 0x1298 0x280000400 rc1=0 and rc2=0 PASS 93 (16s) == sanityn test 94: signal vs CP callback race =========== 00:46:27 (1790743587) osc.lustre-OST0000-osc-ffff8acb843b7800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8acb9097b000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8acb843b7800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8acb9097b000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.210421 s, 38.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.02969 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 289257 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0581521 s, 70.4 kB/s osc.lustre-OST0000-osc-ffff8acb843b7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8acb9097b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb843b7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acb9097b000.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:46:45 (1790743605) llite.lustre-ffff8acb843b7800.fast_read=0 llite.lustre-ffff8acb9097b000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0640926 s, 32.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8acb843b7800.fast_read=1 llite.lustre-ffff8acb9097b000.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:47:07 (1790743627) llite.lustre-ffff8acb843b7800.fast_read=0 llite.lustre-ffff8acb9097b000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00519851 s, 2.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8acb843b7800.fast_read=1 llite.lustre-ffff8acb9097b000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:47:20 (1790743640) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:47:25 (1790743645) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00409759 s, 1000 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:47:33 (1790743653) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143244 s, 14.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:47:41 (1790743661) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0970209 s, 21.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 00:47:47 (1790743667) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0090009 s, 1.3 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 00:47:55 (1790743675) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00373853 s, 1.1 MB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 00:48:02 (1790743682) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0183029 s, 224 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 00:48:09 (1790743689) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0139349 s, 294 kB/s PASS 101c (8s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 00:48:17 (1790743697) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (6s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 00:48:23 (1790743703) 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) 192.168.201.121@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.201.121@tcp:/lustre /mnt/lustre Starting client: oleg121-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.201.121@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.201.121@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 PASS 102b (18s) == sanityn test 103: Test size correctness with lockahead ========================================================== 00:48:41 (1790743721) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790743724 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790743728 Finishing test test23 at 1790743728 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 00:48:56 (1790743736) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (5s) == sanityn test 105: Glimpse and lock cancel race ======== 00:49:00 (1790743740) fail_loc=0x416 PASS 105 (19s) == sanityn test 106a: Verify the btime via statx() ======= 00:49:19 (1790743759) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:49:23 (1790743763) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0360219 s, 28.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 00:49:25.000000000 -0400 Modify: 2026-09-30 00:49:25.000000000 -0400 Change: 2026-09-30 00:49:25.000000000 -0400 Birth: 2026-09-30 00:49:24.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1790743764 1790743765 1790743765 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 00:49:25.000000000 -0400 Modify: 2026-09-30 00:49:25.000000000 -0400 Change: 2026-09-30 00:49:25.000000000 -0400 Birth: 2026-09-30 00:49:24.000000000 -0400 1790743765 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 00:49:32 (1790743772) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 00:49:39 (1790743779) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.273115 s, 38.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4806 0x12c6 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:49:48 (1790743788) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294932 s, 35.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4767 0x129f 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 00:50:03 (1790743803) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0100239 s, 998 kB/s llite.lustre-ffff8acb83534000.hybrid_io=0 llite.lustre-ffff8acb8573c800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.64737 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:50:15 (1790743815) 192.168.201.121@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.121@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: kmemleak disabled oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre 192.168.201.121@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: kmemleak disabled oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 PASS 109 (69s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:51:24 (1790743884) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 00:51:27.000000000 -0400 Modify: 2026-09-30 00:51:27.000000000 -0400 Change: 2026-09-30 00:51:27.000000000 -0400 Birth: 2026-09-30 00:51:27.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg121-client.virtnet oleg121-client.virtnet PASS 110 (173s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:54:17 (1790744057) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 00:54:21 (1790744061) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 00:54:25 (1790744065) lustre-OST0000-osc-ffff8acb84eb2800 active. lustre-OST0001-osc-ffff8acb84eb2800 active. lustre-OST0000-osc-ffff8acb84eb2800 active. lustre-OST0001-osc-ffff8acb84eb2800 active. MGC192.168.201.121@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 00:54:34 (1790744074) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:54:38 (1790744078) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:54:41 (1790744081) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 00:54:44 (1790744084) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0122729 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0946762 s, 216 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 00:54:53 (1790744093) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:55:03 (1790744103) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2331 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3048 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.291 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2725 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3689 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2513 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2302 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1754 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2123 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2603 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.286 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3207 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3135 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2728 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2888 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3781 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3124 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3727 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8acb84eb2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acb84eb2800.ost_server_uuid in FULL state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 9863 sec ============== 00:56:28 (1790744188) === sanityn: start cleanup 00:56:30 (1790744190) === Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg121-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:00:04 (1790744404) ===