-----============= acceptance-small: sanityn ============----- Fri Sep 11 21:09:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg429-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 21:09:28 (1789175368) === Starting client oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Mount client oleg429-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Started clients oleg429-client.virtnet: 192.168.204.129@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg429-client.virtnet: executing check_config_client /mnt/lustre oleg429-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg429-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ddc102cd000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8ddc1063b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc102cd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc1063b800.idle_timeout=debug disable quota as required oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:10:07 (1789175407) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027605 s, 38.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7642] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:10:10 (1789175410) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:10:19 (1789175419) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:10:27 (1789175427) -rw-r--r-- 1 root root 0 Sep 11 21:10 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:10:34 (1789175434) -rw-r--r-- 1 root root 0 Sep 11 21:10 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:10:41 (1789175441) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:10:48 (1789175448) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:10:55 (1789175455) total 1 -rwxrwxrwx 1 root root 0 Sep 11 21:10 f2a -rwxrwxrwx 1 root root 0 Sep 11 21:10 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 11 21:10 f2a -rwxrwxrwx 1 root root 0 Sep 11 21:10 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0105799 s, 48.4 kB/s PASS 2e (10s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:11:05 (1789175465) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 21:11:08 (1789175468) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.410966 s, 5.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:11:17 (1789175477) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:11:24 (1789175484) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:11:32 (1789175492) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 21:11:39 (1789175499) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:11:47 (1789175507) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:11:55 (1789175515) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:12:02 (1789175522) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 21:12:11 (1789175531) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855493 s, 12.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.110494 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:12:20 (1789175540) 1+0 records in 1+0 records out 1 byte copied, 0.00926466 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0142818 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00983044 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111779 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010002 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0215568 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172528 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0379663 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0264877 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0153942 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.012233 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0157008 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:12:29 (1789175549) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0211163 s, 145 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0541628 s, 75.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000261883 s, 11.7 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:12:36 (1789175556) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:12:44 (1789175564) warning: '-runtime' deprecated, use '-t runtime' instead - link 298 (time 1789175576.75 total 10.05 last 29.66) - link 542 (time 1789175586.75 total 20.05 last 24.40) - link 862 (time 1789175596.75 total 30.05 last 31.98) - link 1120 (time 1789175606.77 total 40.07 last 25.76) - link 1400 (time 1789175616.78 total 50.09 last 27.95) - link 1697 (time 1789175626.79 total 60.09 last 29.70) - link 1968 (time 1789175636.80 total 70.10 last 27.06) - link 2232 (time 1789175646.84 total 80.14 last 26.29) - link 2511 (time 1789175656.89 total 90.19 last 27.78) total: 2789 link in 100.02 seconds: 27.88 ops/second using seed 1033489242 running for 100 seconds - stat 4024 (time 1789175578 ; total 11 ; last 11) - stat 9226 (time 1789175589 ; total 22 ; last 11) - stat 10000 (time 1789175590 ; total 23 ; last 1) - stat 14727 (time 1789175601 ; total 34 ; last 11) - stat 20000 (time 1789175611 ; total 44 ; last 10) - stat 23447 (time 1789175622 ; total 55 ; last 11) - stat 25743 (time 1789175633 ; total 66 ; last 11) - stat 27776 (time 1789175644 ; total 77 ; last 11) - stat 29986 (time 1789175655 ; total 88 ; last 11) - stat 30000 (time 1789175655 ; total 88 ; last 0) - stat 32037 (time 1789175666 ; total 99 ; last 11) total: 32276 stats in 100 seconds: 322.760010 stats/second - unlinked 0 (time 1789175617 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2789) error: No such file or directory total: 2788 unlinks in 88 seconds: 31.681818 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21446) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21448) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 21:15:12 (1789175712) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:15:22 (1789175722) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5147: 22633 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:15:30 (1789175730) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5147: 23212 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:15:38 (1789175738) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23791 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:15:46 (1789175746) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24373 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:15:53 (1789175753) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24956 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:16:02 (1789175762) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7529472 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7529472KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:16:58 (1789175818) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.603914 s, 13.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.702627 s, 11.9 MB/s Chance of close/open is 1 in 50 Seed set to 5824 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x276aaf 000001[0] 1789175825.607587 trunc from 00000000 to 0x276aae (0x276aaf bytes) 000003[1] 1789175825.678467 trunc from 0x276aaf to 0x02711d (0x24f991 bytes) 000004[1] 1789175825.721724 mapwrite 0x6847d0 thru 0x68a955 (0x6186 bytes) 000006[0] 1789175825.821252 mapread 0x4e265f thru 0x4e4465 (0x1e07 bytes) 000010[0] 1789175825.971501 punch from 0x01bd91 to 0x0223a0 (0x6610 bytes) 000011[1] 1789175825.990989 mapread 0x1d57b5 thru 0x1e4925 (0xf171 bytes) 000012[1] 1789175826.101489 mapwrite 0x874cca thru 0x877f9c (0x32d3 bytes) 000013[0] 1789175826.294288 mapwrite 0x9021e5 thru 0x90734b (0x5167 bytes) truncating to largest ever: 0x964472 000014[1] 1789175826.474110 trunc from 0x90734c to 0x964471 (0x5d126 bytes) 000016[0] 1789175826.566958 trunc from 0x964472 to 0x1f5017 (0x76f45a bytes) 000019[0] 1789175826.754372 mapwrite 0x945dfe thru 0x9530a5 (0xd2a8 bytes) 000020[0] 1789175827.102209 mapwrite 0x6b7e94 thru 0x6ba685 (0x27f2 bytes) 000026[0] 1789175827.510362 mapwrite 0x7aa308 thru 0x7b068c (0x6385 bytes) 000027[0] 1789175827.764064 read 0x0a5198 thru 0x0a835d (0x31c6 bytes) 000028[0] 1789175827.779363 trunc from 0x9530a6 to 0x0a9a41 (0x8a9664 bytes) 000030[0] 1789175827.944484 mapread 0x0543d3 thru 0x05c75f (0x838d bytes) 000031[1] 1789175828.066515 mapwrite 0x0893c4 thru 0x097474 (0xe0b1 bytes) 000032[0] 1789175828.354563 punch from 0x0940a0 to 0x09d483 (0x93e4 bytes) 000033[0] 1789175828.430713 mapwrite 0x2e7837 thru 0x2f46b1 (0xce7b bytes) 000034[0] 1789175828.595625 mapread 0x0063e8 thru 0x014573 (0xe18c bytes) 000035[1] 1789175828.739021 trunc from 0x2f46b2 to 0x1a999c (0x14ad15 bytes) 000036[1] 1789175828.860285 punch from 0x131f4b to 0x13572b (0x37e1 bytes) 000037[1] 1789175828.942980 punch from 0x14fe71 to 0x158b9b (0x8d2b bytes) 000040[1] 1789175829.050722 mapwrite 0x3c0f5d thru 0x3c3b50 (0x2bf4 bytes) 000042[0] 1789175829.263514 mapwrite 0x3e44dd thru 0x3e9f83 (0x5aa7 bytes) 000045[1] 1789175829.589160 mapread 0x2b8cf8 thru 0x2bef0c (0x6215 bytes) 000047[1] 1789175829.699210 read 0x09fd92 thru 0x0af3e0 (0xf64f bytes) 000048[1] 1789175829.739465 punch from 0x17e7e0 to 0x18d2e0 (0xeb01 bytes) 000049[0] 1789175829.823647 read 0x10e555 thru 0x110767 (0x2213 bytes) 000051[1] 1789175830.002814 trunc from 0x3e9f84 to 0x47b967 (0x919e4 bytes) 000052[0] 1789175830.146229 read 0x28a153 thru 0x297f93 (0xde41 bytes) 000055[1] 1789175830.213813 trunc from 0x47b968 to 0x549532 (0xcdbcb bytes) 000057[0] 1789175830.331396 punch from 0x05746f to 0x05bea0 (0x4a32 bytes) 000060[1] 1789175830.474872 trunc from 0x549533 to 0x724d61 (0x1db82f bytes) 000062[0] 1789175830.670655 mapwrite 0x095e18 thru 0x09c482 (0x666b bytes) 000063[0] 1789175830.866288 write 0x39e60d thru 0x3a0687 (0x207b bytes) 000065[1] 1789175830.962246 write 0x675c32 thru 0x6772c6 (0x1695 bytes) 000066[1] 1789175831.022360 mapwrite 0x278c73 thru 0x280154 (0x74e2 bytes) 000067[0] 1789175831.180274 write 0x06a911 thru 0x073465 (0x8b55 bytes) 000068[0] 1789175831.283991 punch from 0x229eee to 0x22bfaa (0x20bd bytes) 000070[1] 1789175831.337953 mapwrite 0x9803d7 thru 0x9825bb (0x21e5 bytes) 000071[1] 1789175831.551325 punch from 0x6265c7 to 0x62ec0d (0x8647 bytes) 000072[1] 1789175831.585640 read 0x5ec124 thru 0x5fb832 (0xf70f bytes) 000073[1] 1789175831.625978 punch from 0x53e11c to 0x548638 (0xa51d bytes) 000077[0] 1789175831.702624 mapread 0x257af0 thru 0x25dfff (0x6510 bytes) 000078[0] 1789175831.775264 mapread 0x895fd1 thru 0x89d9f0 (0x7a20 bytes) 000079[0] 1789175831.831386 mapwrite 0x464844 thru 0x468e5a (0x4617 bytes) 000080[0] 1789175832.010482 punch from 0x2da781 to 0x2e9904 (0xf184 bytes) 000083[0] 1789175832.130438 trunc from 0x9825bc to 0x425476 (0x55d145 bytes) 000084[1] 1789175832.162419 mapwrite 0x6ed8e7 thru 0x6f5128 (0x7842 bytes) 000086[0] 1789175832.366588 punch from 0x553945 to 0x55da6c (0xa128 bytes) 000090[1] 1789175832.453080 trunc from 0x6f5129 to 0x2a0e58 (0x4542d0 bytes) 000091[1] 1789175832.510829 mapread 0x22b97a thru 0x2364a9 (0xab30 bytes) 000093[0] 1789175832.581359 punch from 0x0d6c1c to 0x0d72e4 (0x6c9 bytes) 000094[0] 1789175832.636329 mapread 0x1e2d4e thru 0x1e3043 (0x2f6 bytes) 000095[1] 1789175832.722395 mapread 0x18ad67 thru 0x18b6d2 (0x96c bytes) 000096[0] 1789175832.776491 read 0x07a487 thru 0x0885da (0xe154 bytes) 000097[1] 1789175832.876849 punch from 0x160a22 to 0x16b974 (0xaf53 bytes) 000098[0] 1789175832.918293 mapread 0x09682b thru 0x0a1ecd (0xb6a3 bytes) 000100[1] 1789175833.063569 read 0x0a0659 thru 0x0afc33 (0xf5db bytes) 000106[0] 1789175833.222187 trunc from 0x2a0e59 to 0x322e55 (0x81ffd bytes) 000107[0] 1789175833.334383 mapread 0x2bf14c thru 0x2c4498 (0x534d bytes) 000110[1] 1789175833.418370 trunc from 0x322e56 to 0x2b48b1 (0x6e5a4 bytes) 000111[1] 1789175833.478935 trunc from 0x2b48b2 to 0x2931c8 (0x216e9 bytes) 000112[1] 1789175833.561454 punch from 0x0f4098 to 0x0f659d (0x2506 bytes) 000116[1] 1789175833.586659 write 0x7555d1 thru 0x7626af (0xd0df bytes) 000117[1] 1789175833.643881 read 0x72019d thru 0x72a299 (0xa0fd bytes) 000123[0] 1789175833.800962 read 0x29b488 thru 0x2a19dc (0x6555 bytes) 000125[0] 1789175833.936196 trunc from 0x7626b0 to 0x4cab30 (0x297b7f bytes) 000126[0] 1789175834.075825 read 0x191b11 thru 0x1931d9 (0x16c9 bytes) 000128[1] 1789175834.094548 read 0x03c3ce thru 0x0480d1 (0xbd04 bytes) 000129[1] 1789175834.130348 trunc from 0x4cab31 to 0x95c32f (0x4917ff bytes) 000130[0] 1789175834.209671 write 0x10ec83 thru 0x10f378 (0x6f6 bytes) 000132[0] 1789175834.352410 write 0x0fa094 thru 0x107b1b (0xda88 bytes) 000133[0] 1789175834.426540 trunc from 0x95c330 to 0x27f730 (0x6dcbff bytes) 000134[1] 1789175834.511882 mapwrite 0x90139c thru 0x909917 (0x857c bytes) 000135[1] 1789175834.753624 punch from 0x0c14f0 to 0x0cef7c (0xda8d bytes) 000140[1] 1789175834.787396 write 0x26ae53 thru 0x26deb2 (0x3060 bytes) 000142[0] 1789175834.797876 punch from 0x1c6f9e to 0x1cf982 (0x89e5 bytes) 000146[1] 1789175834.916965 read 0x616256 thru 0x61d4b4 (0x725f bytes) 000147[1] 1789175834.945512 write 0x722312 thru 0x728eb3 (0x6ba2 bytes) 000149[1] 1789175834.979279 trunc from 0x909918 to 0x1e8b59 (0x720dbe bytes) 000151[1] 1789175835.060647 mapread 0x1c8b7c thru 0x1d3d93 (0xb218 bytes) 000156[0] 1789175835.164739 mapwrite 0x1bafb3 thru 0x1bee94 (0x3ee2 bytes) 000157[0] 1789175835.352495 read 0x1d0663 thru 0x1d0fb4 (0x952 bytes) 000158[0] 1789175835.420853 read 0x1a2fcb thru 0x1a7cd5 (0x4d0b bytes) 000163[0] 1789175835.521213 trunc from 0x1e8b5a to 0x19a247 (0x4e912 bytes) 000165[1] 1789175835.583684 punch from 0x0e56e5 to 0x0f4e22 (0xf73e bytes) 000167[0] 1789175835.636749 trunc from 0x19a248 to 0x2531b7 (0xb8f70 bytes) 000169[1] 1789175835.709465 mapwrite 0x0fe656 thru 0x10a743 (0xc0ee bytes) 000172[0] 1789175835.918401 trunc from 0x2531b8 to 0x302ee3 (0xafd2c bytes) 000173[1] 1789175835.994995 write 0x9a438f thru 0x9a65c7 (0x2239 bytes) 000174[1] 1789175836.040306 trunc from 0x9a65c8 to 0x91c297 (0x8a330 bytes) 000175[1] 1789175836.092643 mapread 0x3aa545 thru 0x3af67c (0x5138 bytes) 000176[1] 1789175836.169892 punch from 0x68eceb to 0x68f229 (0x53f bytes) 000178[1] 1789175836.236824 read 0x4f34b9 thru 0x4f7b78 (0x46c0 bytes) 000179[1] 1789175836.321805 read 0x597401 thru 0x59ee6d (0x7a6d bytes) 000180[1] 1789175836.335804 write 0x41273c thru 0x413e2a (0x16ef bytes) 000183[0] 1789175836.383946 mapread 0x3eea47 thru 0x3f15a3 (0x2b5d bytes) 000187[1] 1789175836.659228 mapwrite 0x96e6a2 thru 0x97e39a (0xfcf9 bytes) 000190[0] 1789175837.013804 trunc from 0x97e39b to 0x1932ca (0x7eb0d0 bytes) 000191[1] 1789175837.081406 write 0x5694ec thru 0x57175a (0x826f bytes) 000192[0] 1789175837.140801 trunc from 0x57175b to 0x038fdb (0x53877f bytes) 000195[1] 1789175837.283851 read 0x01a631 thru 0x021f51 (0x7921 bytes) 000197[0] 1789175837.312830 mapread 0x022ea5 thru 0x027b7f (0x4cdb bytes) 000199[1] 1789175837.403293 write 0x6b00f4 thru 0x6b5c3f (0x5b4c bytes) 000200[0] 1789175837.469745 punch from 0x0cff64 to 0x0d93f2 (0x948f bytes) 000201[1] 1789175837.536779 write 0x62b6cc thru 0x62c6ff (0x1034 bytes) 000203[1] 1789175837.594227 punch from 0x046cd8 to 0x049fe7 (0x3310 bytes) 000205[1] 1789175837.639798 write 0x937a64 thru 0x93fbb5 (0x8152 bytes) 000208[0] 1789175837.665832 punch from 0x72f45b to 0x73f1a5 (0xfd4b bytes) 000210[0] 1789175837.732000 read 0x4d2525 thru 0x4de35d (0xbe39 bytes) 000211[0] 1789175837.769570 mapwrite 0x8150fe thru 0x818a3e (0x3941 bytes) 000213[1] 1789175837.952712 trunc from 0x93fbb6 to 0x49cf71 (0x4a2c44 bytes) truncating to largest ever: 0x994c9d 000214[1] 1789175838.047371 trunc from 0x49cf72 to 0x994c9c (0x4f7d2b bytes) 000215[1] 1789175838.105872 mapread 0x2575d9 thru 0x25e20d (0x6c35 bytes) 000217[0] 1789175838.286746 read 0x619e99 thru 0x62714c (0xd2b4 bytes) 000218[1] 1789175838.331526 write 0x1cb83f thru 0x1d9fb2 (0xe774 bytes) 000219[1] 1789175838.352809 read 0x8265d6 thru 0x826a3e (0x469 bytes) 000221[1] 1789175838.373493 trunc from 0x994c9d to 0x95297b (0x42321 bytes) 000223[0] 1789175838.488380 punch from 0x2f1c40 to 0x2fd12f (0xb4f0 bytes) 000227[0] 1789175838.707462 punch from 0x8eaa05 to 0x8ef34f (0x494b bytes) 000229[1] 1789175838.758003 punch from 0x3ff58c to 0x40ef13 (0xf988 bytes) 000232[1] 1789175838.881007 punch from 0x43f8e5 to 0x449685 (0x9da1 bytes) 000234[0] 1789175838.979862 write 0x3a02e5 thru 0x3a4eae (0x4bca bytes) 000236[0] 1789175839.089718 punch from 0x13bd37 to 0x145505 (0x97cf bytes) 000239[0] 1789175839.231766 read 0x49daef thru 0x4a0814 (0x2d26 bytes) 000241[1] 1789175839.273397 mapwrite 0x6ff6e3 thru 0x70c89b (0xd1b9 bytes) 000242[0] 1789175839.441777 write 0x7a512a thru 0x7b214a (0xd021 bytes) 000243[0] 1789175839.478934 write 0x94be0f thru 0x95abf1 (0xede3 bytes) 000244[0] 1789175839.533168 read 0x46e7e7 thru 0x479de0 (0xb5fa bytes) 000247[1] 1789175839.680238 write 0x53429c thru 0x5430e1 (0xee46 bytes) 000249[0] 1789175839.811688 trunc from 0x95abf2 to 0x34c53a (0x60e6b7 bytes) 000252[1] 1789175839.994879 mapwrite 0x145441 thru 0x152622 (0xd1e2 bytes) 000255[1] 1789175840.473214 punch from 0x1f680d to 0x1f7c96 (0x148a bytes) 000257[1] 1789175840.533706 punch from 0x2431dc to 0x243220 (0x45 bytes) 000261[0] 1789175840.714221 mapread 0x20eed3 thru 0x219adf (0xac0d bytes) 000262[0] 1789175840.837945 mapwrite 0x7c6a2c thru 0x7d67a8 (0xfd7d bytes) 000263[0] 1789175841.113905 read 0x4b9233 thru 0x4b95f3 (0x3c1 bytes) 000267[1] 1789175841.212020 punch from 0x0fcc87 to 0x0feb8e (0x1f08 bytes) 000268[1] 1789175841.301802 trunc from 0x7d67a9 to 0x6d5b77 (0x100c31 bytes) 000269[1] 1789175841.427508 mapread 0x35f5e4 thru 0x361c00 (0x261d bytes) 000270[1] 1789175841.501351 write 0x35b88d thru 0x35ebb5 (0x3329 bytes) 000271[1] 1789175841.511895 read 0x5f35e6 thru 0x5f6706 (0x3121 bytes) 000272[1] 1789175841.557902 read 0x60bf0a thru 0x61bd87 (0xfe7e bytes) 000273[0] 1789175841.589931 mapread 0x584b02 thru 0x5903ab (0xb8aa bytes) 000274[1] 1789175841.707754 mapwrite 0x5548c6 thru 0x563ca3 (0xf3de bytes) 000275[0] 1789175841.887447 write 0x7468f8 thru 0x74ad37 (0x4440 bytes) 000279[0] 1789175841.978486 mapwrite 0x773097 thru 0x781f22 (0xee8c bytes) 000281[0] 1789175842.305711 write 0x138fa5 thru 0x13c2cc (0x3328 bytes) 000284[1] 1789175842.345124 trunc from 0x781f23 to 0x51bbfb (0x266327 bytes) 000285[1] 1789175842.466066 mapwrite 0x401a59 thru 0x403883 (0x1e2b bytes) 000286[1] 1789175842.655343 read 0x42d4f6 thru 0x435659 (0x8164 bytes) 000287[1] 1789175842.703734 mapread 0x4dc72e thru 0x4e2ff7 (0x68ca bytes) 000289[1] 1789175842.794717 mapwrite 0x085f79 thru 0x093af5 (0xdb7d bytes) 000290[1] 1789175842.927914 read 0x46cb70 thru 0x46cdd1 (0x262 bytes) 000291[1] 1789175842.984914 write 0x407760 thru 0x40c04b (0x48ec bytes) 000292[1] 1789175843.046556 write 0x956966 thru 0x9596d6 (0x2d71 bytes) 000293[0] 1789175843.065562 trunc from 0x9596d7 to 0x5f76ba (0x36201c bytes) 000294[0] 1789175843.178733 write 0x6c6c6a thru 0x6d0344 (0x96db bytes) 000297[1] 1789175843.283722 punch from 0x4f29a8 to 0x4f32f1 (0x94a bytes) 000298[0] 1789175843.384690 mapwrite 0x5fb722 thru 0x60640f (0xacee bytes) 000299[0] 1789175843.663897 read 0x16758d thru 0x168bae (0x1622 bytes) 000304[1] 1789175843.680238 mapread 0x2234e5 thru 0x22377d (0x299 bytes) 000305[1] 1789175843.820425 trunc from 0x6d0345 to 0x101848 (0x5ceafc bytes) 000306[0] 1789175843.906408 read 0x02ad4a thru 0x02dd59 (0x3010 bytes) 000307[1] 1789175843.964881 write 0x0e1bf6 thru 0x0e5901 (0x3d0c bytes) 000308[0] 1789175843.987836 write 0x1b03f5 thru 0x1b869e (0x82aa bytes) 000309[1] 1789175844.024708 mapread 0x13ec9e thru 0x144275 (0x55d8 bytes) 000311[0] 1789175844.092971 mapread 0x0cdb13 thru 0x0d2c16 (0x5104 bytes) 000312[1] 1789175844.132185 punch from 0x0a2143 to 0x0a8d9f (0x6c5d bytes) 000313[1] 1789175844.181014 write 0x6ab679 thru 0x6b645e (0xade6 bytes) 000315[0] 1789175844.245978 trunc from 0x6b645f to 0x4933e9 (0x223075 bytes) 000319[0] 1789175844.547222 punch from 0x421889 to 0x42f656 (0xddce bytes) 000321[0] 1789175844.580947 trunc from 0x4933ea to 0x852ebe (0x3bfad5 bytes) 000325[0] 1789175844.785405 write 0x56ed6d thru 0x57898a (0x9c1e bytes) 000326[1] 1789175844.807367 read 0x6a26fd thru 0x6a648c (0x3d90 bytes) 000327[0] 1789175844.857213 mapwrite 0x6e50fc thru 0x6f1090 (0xbf95 bytes) 000328[0] 1789175844.986431 trunc from 0x852ebf to 0x5f810e (0x25adb0 bytes) 000329[1] 1789175845.058637 write 0x54df39 thru 0x554dfa (0x6ec2 bytes) 000330[0] 1789175845.116541 mapwrite 0x9520fd thru 0x957b97 (0x5a9b bytes) 000331[1] 1789175845.419863 write 0x1760b2 thru 0x17a351 (0x42a0 bytes) 000332[0] 1789175845.528770 punch from 0x0f8a60 to 0x0fa267 (0x1808 bytes) 000333[1] 1789175845.576990 write 0x62d494 thru 0x63cc73 (0xf7e0 bytes) 000335[1] 1789175845.705936 punch from 0x6fb28f to 0x705507 (0xa279 bytes) 000337[1] 1789175845.754335 mapwrite 0x6606f8 thru 0x66caf1 (0xc3fa bytes) 000338[1] 1789175845.949863 mapwrite 0x122b68 thru 0x12fef0 (0xd389 bytes) 000340[0] 1789175846.092850 mapwrite 0x68cd2c thru 0x69b543 (0xe818 bytes) 000341[0] 1789175846.267901 read 0x07f53b thru 0x08e0f7 (0xebbd bytes) 000343[0] 1789175846.316844 mapwrite 0x7b19e4 thru 0x7b51b9 (0x37d6 bytes) 000344[0] 1789175846.448194 write 0x2df88c thru 0x2ee14b (0xe8c0 bytes) 000345[0] 1789175846.509444 read 0x08810c thru 0x08b42a (0x331f bytes) 000349[1] 1789175846.518340 mapwrite 0x2842cb thru 0x28990f (0x5645 bytes) 000353[1] 1789175846.870771 trunc from 0x957b98 to 0x06b9cc (0x8ec1cb bytes) 000354[0] 1789175847.033702 punch from 0x01892a to 0x01dce5 (0x53bc bytes) 000355[0] 1789175847.088396 punch from 0x05c8ef to 0x05fa95 (0x31a7 bytes) 000358[1] 1789175847.227055 mapread 0x049b4b thru 0x0586c1 (0xeb77 bytes) 000360[1] 1789175847.387251 punch from 0x064b02 to 0x06b9cc (0x6ecb bytes) 000362[0] 1789175847.430319 mapread 0x025604 thru 0x0282bc (0x2cb9 bytes) 000363[1] 1789175847.523015 write 0x464d7f thru 0x46bcdb (0x6f5d bytes) 000366[1] 1789175847.578769 read 0x46a0b2 thru 0x46bcdb (0x1c2a bytes) 000367[1] 1789175847.579402 read 0x21d1e9 thru 0x21fdd6 (0x2bee bytes) 000368[1] 1789175847.600145 read 0x1ff72e thru 0x1ff995 (0x268 bytes) 000370[1] 1789175847.605836 mapread 0x051637 thru 0x057baa (0x6574 bytes) 000371[0] 1789175847.642962 mapwrite 0x08d20f thru 0x09732e (0xa120 bytes) 000373[1] 1789175848.216669 mapread 0x466d19 thru 0x46bcdb (0x4fc3 bytes) 000374[0] 1789175848.264896 mapwrite 0x38151a thru 0x38db7f (0xc666 bytes) 000377[1] 1789175848.734419 mapwrite 0x65b662 thru 0x667912 (0xc2b1 bytes) 000378[1] 1789175849.016894 read 0x22a9fd thru 0x232140 (0x7744 bytes) 000379[1] 1789175849.089534 read 0x13cf7e thru 0x13d0a9 (0x12c bytes) 000381[1] 1789175849.177533 write 0x515e4c thru 0x520a45 (0xabfa bytes) 000382[0] 1789175849.222525 mapwrite 0x5d62cc thru 0x5de06c (0x7da1 bytes) 000383[0] 1789175849.379959 mapwrite 0x3f599e thru 0x402c1a (0xd27d bytes) 000384[0] 1789175849.663800 mapwrite 0x4b8f08 thru 0x4c192a (0x8a23 bytes) 000385[1] 1789175849.834556 read 0x5ef95c thru 0x5f4230 (0x48d5 bytes) 000386[1] 1789175849.874222 mapread 0x58a7c3 thru 0x594328 (0x9b66 bytes) 000387[1] 1789175849.919923 trunc from 0x667913 to 0x964750 (0x2fce3e bytes) 000389[0] 1789175850.005216 trunc from 0x964751 to 0x3b32f0 (0x5b1460 bytes) truncating to largest ever: 0x9a152d 000390[0] 1789175850.109291 trunc from 0x3b32f1 to 0x9a152c (0x5ee23c bytes) 000393[1] 1789175850.145458 punch from 0x0ca5a2 to 0x0cab6b (0x5ca bytes) 000394[1] 1789175850.208590 mapread 0x967e29 thru 0x96fdfd (0x7fd5 bytes) 000396[0] 1789175850.287398 mapread 0x4afa62 thru 0x4b6586 (0x6b25 bytes) truncating to largest ever: 0x9f8d90 000397[1] 1789175850.328598 trunc from 0x9a152d to 0x9f8d8f (0x57863 bytes) 000398[1] 1789175850.384880 write 0x8a96aa thru 0x8b14f5 (0x7e4c bytes) 000399[1] 1789175850.454852 mapwrite 0x3e7812 thru 0x3ee60d (0x6dfc bytes) 000401[1] 1789175850.591792 trunc from 0x9f8d90 to 0x0baa28 (0x93e367 bytes) 000402[0] 1789175850.633553 mapread 0x030757 thru 0x0339e9 (0x3293 bytes) 000403[1] 1789175850.726646 write 0x317509 thru 0x318853 (0x134b bytes) 000405[0] 1789175850.778544 mapwrite 0x88794a thru 0x8936f4 (0xbdab bytes) 000406[0] 1789175851.036719 punch from 0x08953a to 0x093933 (0xa3fa bytes) 000407[1] 1789175851.078464 mapwrite 0x6846e1 thru 0x69124c (0xcb6c bytes) 000408[1] 1789175851.247816 write 0x63c94d thru 0x63d2d2 (0x986 bytes) 000410[0] 1789175851.333049 read 0x10d55b thru 0x113c52 (0x66f8 bytes) 000412[0] 1789175851.391423 trunc from 0x8936f5 to 0x166068 (0x72d68c bytes) 000413[0] 1789175851.442698 mapwrite 0x5f1ff2 thru 0x5f9a7c (0x7a8b bytes) 000415[1] 1789175851.685051 mapwrite 0x863240 thru 0x872af7 (0xf8b8 bytes) 000416[1] 1789175851.979977 mapread 0x81a9d8 thru 0x81aa00 (0x29 bytes) 000418[0] 1789175852.092220 read 0x27c88e thru 0x2816b5 (0x4e28 bytes) 000419[0] 1789175852.139601 read 0x232768 thru 0x233e6d (0x1706 bytes) 000420[0] 1789175852.146057 punch from 0x33a95b to 0x347d42 (0xd3e8 bytes) 000422[1] 1789175852.218527 mapwrite 0x2e041b thru 0x2f00a4 (0xfc8a bytes) 000424[1] 1789175852.411818 write 0x85a33f thru 0x85f117 (0x4dd9 bytes) 000429[1] 1789175852.595346 trunc from 0x872af8 to 0x9af408 (0x13c911 bytes) 000431[1] 1789175852.763046 read 0x22b3be thru 0x23a617 (0xf25a bytes) 000432[1] 1789175852.845380 read 0x7eec23 thru 0x7f6e03 (0x81e1 bytes) 000433[0] 1789175852.901990 trunc from 0x9af409 to 0x686e6f (0x328599 bytes) 000436[0] 1789175853.283023 read 0x18f464 thru 0x19a885 (0xb422 bytes) 000437[1] 1789175853.356311 write 0x50f4c2 thru 0x5154d9 (0x6018 bytes) 000438[0] 1789175853.450234 trunc from 0x686e70 to 0x9e55ee (0x35e77f bytes) 000439[1] 1789175853.537929 mapread 0x2ffa2a thru 0x30072b (0xd02 bytes) 000440[0] 1789175853.642015 punch from 0x3acb62 to 0x3ae23b (0x16da bytes) 000442[0] 1789175853.790344 punch from 0x50c1db to 0x5183e7 (0xc20d bytes) 000444[1] 1789175853.896903 punch from 0x3b2194 to 0x3b7e65 (0x5cd2 bytes) 000445[1] 1789175853.977912 punch from 0x685480 to 0x6935fb (0xe17c bytes) 000446[0] 1789175854.081686 write 0x414e52 thru 0x424e34 (0xffe3 bytes) 000447[0] 1789175854.176794 write 0x178f2c thru 0x17947f (0x554 bytes) 000448[1] 1789175854.216369 trunc from 0x9e55ef to 0x22f444 (0x7b61aa bytes) 000449[0] 1789175854.311117 trunc from 0x22f445 to 0x1de0f2 (0x51352 bytes) 000450[1] 1789175854.372628 read 0x10b6fb thru 0x110314 (0x4c1a bytes) 000453[1] 1789175854.512272 trunc from 0x1de0f3 to 0x6e9a81 (0x50b98f bytes) 000454[0] 1789175854.581420 read 0x3e29f1 thru 0x3ec0b5 (0x96c5 bytes) 000455[1] 1789175854.620316 read 0x492bf4 thru 0x497741 (0x4b4e bytes) 000456[1] 1789175854.639930 punch from 0x0939c3 to 0x09687f (0x2ebd bytes) 000457[1] 1789175854.677911 punch from 0x25de6d to 0x263036 (0x51ca bytes) 000461[0] 1789175854.762915 trunc from 0x6e9a82 to 0x64b79c (0x9e2e5 bytes) 000464[1] 1789175854.892716 mapread 0x1f47b9 thru 0x20297b (0xe1c3 bytes) 000466[1] 1789175855.018400 mapread 0x412235 thru 0x41ab9d (0x8969 bytes) 000467[1] 1789175855.158319 mapread 0x404f9c thru 0x40cee6 (0x7f4b bytes) 000468[0] 1789175855.181445 mapread 0x1afad1 thru 0x1b229c (0x27cc bytes) 000469[1] 1789175855.227801 mapread 0x125302 thru 0x134e69 (0xfb68 bytes) 000470[1] 1789175855.299377 read 0x1b4d57 thru 0x1bd36d (0x8617 bytes) 000471[0] 1789175855.314699 mapread 0x1d36bf thru 0x1e11d2 (0xdb14 bytes) 000472[0] 1789175855.333747 read 0x1b378e thru 0x1c2c36 (0xf4a9 bytes) 000473[0] 1789175855.345656 trunc from 0x64b79d to 0x47f2b3 (0x1cc4e9 bytes) 000474[1] 1789175855.449576 write 0x04f14e thru 0x05564f (0x6502 bytes) 000475[0] 1789175855.506731 read 0x3ff25b thru 0x4009cb (0x1771 bytes) 000478[1] 1789175855.575418 mapread 0x15e74c thru 0x15f09e (0x953 bytes) 000479[1] 1789175855.628001 mapwrite 0x0e7d50 thru 0x0ebf31 (0x41e2 bytes) 000480[1] 1789175855.857478 punch from 0x42a90b to 0x434106 (0x97fc bytes) 000483[0] 1789175855.906420 punch from 0x2eba02 to 0x2eda2e (0x202d bytes) 000486[0] 1789175855.986612 read 0x154fef thru 0x15e45f (0x9471 bytes) 000487[1] 1789175856.013147 write 0x7a1502 thru 0x7a7886 (0x6385 bytes) 000489[1] 1789175856.054853 read 0x190797 thru 0x196793 (0x5ffd bytes) 000490[1] 1789175856.097700 mapread 0x4ec3ee thru 0x4f2da4 (0x69b7 bytes) 000495[0] 1789175856.170417 punch from 0x6d4039 to 0x6dce94 (0x8e5c bytes) 000497[1] 1789175856.239642 punch from 0x2071d0 to 0x208d17 (0x1b48 bytes) 000498[1] 1789175856.278925 mapread 0x157b74 thru 0x15e3c4 (0x6851 bytes) 000499[1] 1789175856.347450 mapwrite 0x72d13b thru 0x737ebf (0xad85 bytes) 000500[1] 1789175856.469999 read 0x2e0b85 thru 0x2e811d (0x7599 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 5856 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789175857.531979 mapwrite 0x62f000 thru 0x63e5b7 (0xf5b8 bytes) 000002[1] 1789175857.771206 read_OD 0x5a1000 thru 0x5a2fff (0x2000 bytes) 000003[0] 1789175857.843848 write 0x371000 thru 0x372682 (0x1683 bytes) 000004[1] 1789175857.918622 mapread 0x01f000 thru 0x02e973 (0xf974 bytes) 000006[1] 1789175858.040744 mapread 0x3de000 thru 0x3e2e66 (0x4e67 bytes) 000007[0] 1789175858.069467 write 0x41b000 thru 0x428863 (0xd864 bytes) 000010[0] 1789175858.134360 mapread 0x12f000 thru 0x130286 (0x1287 bytes) 000011[0] 1789175858.155353 mapwrite 0x210000 thru 0x214808 (0x4809 bytes) 000012[1] 1789175858.298444 mapwrite 0x00a000 thru 0x018b80 (0xeb81 bytes) 000014[0] 1789175858.451674 read 0x440000 thru 0x44b069 (0xb06a bytes) truncating to largest ever: 0x9af52b 000015[1] 1789175858.486727 trunc from 0x63e5b8 to 0x9af52a (0x370f73 bytes) 000017[1] 1789175858.554541 trunc from 0x9af52b to 0x997a2b (0x17aff bytes) 000018[0] 1789175858.579357 write 0x54e000 thru 0x55c045 (0xe046 bytes) 000019[1] 1789175858.612078 punch from 0x516bfc to 0x51c81d (0x5c22 bytes) 000021[1] 1789175858.659694 mapread 0x12f000 thru 0x139ccb (0xaccc bytes) 000025[1] 1789175858.754075 write_OD 0x3f3000 thru 0x401fff (0xf000 bytes) 000026[1] 1789175858.958605 read_OD 0x412000 thru 0x420fff (0xf000 bytes) 000027[1] 1789175858.977026 mapread 0x21b000 thru 0x2201ee (0x51ef bytes) 000028[0] 1789175859.028226 read 0x433000 thru 0x43aa9b (0x7a9c bytes) 000029[1] 1789175859.043784 write_OD 0x671000 thru 0x67dfff (0xd000 bytes) 000030[0] 1789175859.257017 trunc from 0x997a2c to 0x88703e (0x1109ed bytes) 000032[1] 1789175859.408494 punch from 0x2d7bc7 to 0x2e79f9 (0xfe33 bytes) 000033[1] 1789175859.471950 write_OD 0x03d000 thru 0x042fff (0x6000 bytes) 000037[1] 1789175859.606401 trunc from 0x88703f to 0x8f56a7 (0x6e669 bytes) 000040[1] 1789175859.709412 trunc from 0x8f56a8 to 0x245894 (0x6afe13 bytes) 000041[1] 1789175859.761575 mapread 0x132000 thru 0x132ec4 (0xec5 bytes) 000042[1] 1789175859.777071 mapwrite 0x2cb000 thru 0x2d041d (0x541e bytes) 000043[1] 1789175859.956212 read_OD 0x0cf000 thru 0x0d2fff (0x4000 bytes) 000044[1] 1789175860.009477 trunc from 0x2d041e to 0x09dbac (0x232871 bytes) 000045[1] 1789175860.169713 punch from 0x02a5fc to 0x02ddb4 (0x37b9 bytes) 000046[1] 1789175860.191896 mapwrite 0x294000 thru 0x29be1b (0x7e1c bytes) 000047[0] 1789175860.368813 mapread 0x29b000 thru 0x29bd99 (0xd9a bytes) 000049[1] 1789175860.417684 mapread 0x021000 thru 0x023d36 (0x2d37 bytes) 000050[1] 1789175860.433147 mapwrite 0x29f000 thru 0x2a3624 (0x4625 bytes) 000056[0] 1789175860.601549 write 0x401000 thru 0x406353 (0x5354 bytes) 000059[1] 1789175860.666592 read_OD 0x248000 thru 0x248fff (0x1000 bytes) 000061[0] 1789175860.723936 trunc from 0x406354 to 0x6d303b (0x2ccce8 bytes) 000062[0] 1789175860.761015 mapread 0x4a4000 thru 0x4b3bfc (0xfbfd bytes) 000064[1] 1789175860.826905 trunc from 0x6d303c to 0x554161 (0x17eeda bytes) 000065[0] 1789175860.860887 punch from 0x03973c to 0x044689 (0xaf4e bytes) 000066[0] 1789175860.879265 mapread 0x115000 thru 0x1164e1 (0x14e2 bytes) 000067[1] 1789175860.906383 write_OD 0x70e000 thru 0x718fff (0xb000 bytes) 000070[1] 1789175860.970928 read_OD 0x66b000 thru 0x66cfff (0x2000 bytes) 000072[1] 1789175860.981609 mapwrite 0x1a5000 thru 0x1a9722 (0x4723 bytes) 000074[1] 1789175861.169245 trunc from 0x719000 to 0x20aa40 (0x50e5bf bytes) 000075[0] 1789175861.197880 mapread 0x1c3000 thru 0x1c43d5 (0x13d6 bytes) 000077[1] 1789175861.273689 read_OD 0x04d000 thru 0x055fff (0x9000 bytes) 000078[0] 1789175861.284643 mapread 0x144000 thru 0x14cb24 (0x8b25 bytes) 000080[0] 1789175861.304675 read 0x0a0000 thru 0x0a75d0 (0x75d1 bytes) 000081[1] 1789175861.329698 mapread 0x110000 thru 0x114d5b (0x4d5c bytes) 000083[1] 1789175861.366355 mapwrite 0x16d000 thru 0x16fbeb (0x2bec bytes) 000084[1] 1789175861.582191 punch from 0x0104a6 to 0x01886c (0x83c7 bytes) 000087[1] 1789175861.632928 read_OD 0x1e1000 thru 0x1e6fff (0x6000 bytes) 000088[1] 1789175861.659856 read_OD 0x052000 thru 0x05cfff (0xb000 bytes) 000091[1] 1789175861.676904 mapwrite 0x3ec000 thru 0x3fbb53 (0xfb54 bytes) 000092[0] 1789175861.874333 write 0x699000 thru 0x6a0762 (0x7763 bytes) 000093[0] 1789175861.940910 trunc from 0x6a0763 to 0x3a2d70 (0x2fd9f2 bytes) 000097[1] 1789175862.032619 write_OD 0x6cd000 thru 0x6d0fff (0x4000 bytes) 000098[1] 1789175862.099294 mapwrite 0x8c0000 thru 0x8c54e2 (0x54e3 bytes) 000099[0] 1789175862.407247 mapread 0x334000 thru 0x3352f9 (0x12fa bytes) 000100[0] 1789175862.541190 mapread 0x5fa000 thru 0x605c69 (0xbc6a bytes) 000101[1] 1789175862.567925 trunc from 0x8c54e3 to 0x304ad1 (0x5c0a11 bytes) 000103[0] 1789175862.684834 mapwrite 0x290000 thru 0x294597 (0x4598 bytes) 000104[0] 1789175862.962954 trunc from 0x304ad2 to 0x17a3d6 (0x18a6fb bytes) 000105[0] 1789175862.991394 mapwrite 0x5dc000 thru 0x5e41e2 (0x81e3 bytes) 000107[1] 1789175863.224763 write_OD 0x1c2000 thru 0x1c2fff (0x1000 bytes) 000109[1] 1789175863.337044 mapwrite 0x83e000 thru 0x84648e (0x848f bytes) 000110[0] 1789175863.626295 punch from 0x425f12 to 0x42d8e2 (0x79d1 bytes) 000111[1] 1789175863.692345 read_OD 0x57a000 thru 0x57afff (0x1000 bytes) 000113[0] 1789175863.736771 mapwrite 0x9df000 thru 0x9e93e8 (0xa3e9 bytes) 000115[0] 1789175864.153854 trunc from 0x9e93e9 to 0x49c384 (0x54d064 bytes) 000116[1] 1789175864.260453 punch from 0x1c8776 to 0x1d5d31 (0xd5bc bytes) 000117[0] 1789175864.348827 trunc from 0x49c385 to 0x8827b9 (0x3e6435 bytes) 000119[0] 1789175864.415508 read 0x2ff000 thru 0x309245 (0xa246 bytes) 000120[0] 1789175864.429077 mapwrite 0x485000 thru 0x48a8ee (0x58ef bytes) 000123[0] 1789175864.589800 write 0x75e000 thru 0x768dfc (0xadfd bytes) 000124[1] 1789175864.613319 mapread 0x427000 thru 0x42a0d8 (0x30d9 bytes) 000126[0] 1789175864.771023 mapwrite 0x8bf000 thru 0x8c77ac (0x87ad bytes) 000129[1] 1789175865.001665 mapwrite 0x454000 thru 0x457bc0 (0x3bc1 bytes) 000131[0] 1789175865.167764 trunc from 0x8c77ad to 0x50e10c (0x3b96a0 bytes) truncating to largest ever: 0x9f7af2 000133[1] 1789175865.276597 trunc from 0x50e10d to 0x9f7af1 (0x4e99e5 bytes) 000134[1] 1789175865.347928 mapread 0x318000 thru 0x3224c0 (0xa4c1 bytes) 000135[0] 1789175865.467454 punch from 0x8e7d9f to 0x8ecbe1 (0x4e43 bytes) 000136[0] 1789175865.509300 read 0x300000 thru 0x306dfb (0x6dfc bytes) 000137[1] 1789175865.553424 mapwrite 0x506000 thru 0x512157 (0xc158 bytes) 000138[1] 1789175865.892928 mapread 0x7f0000 thru 0x7fb86e (0xb86f bytes) 000142[0] 1789175866.125514 write 0x789000 thru 0x78fe11 (0x6e12 bytes) 000143[0] 1789175866.223850 read 0x9b1000 thru 0x9b580a (0x480b bytes) 000146[1] 1789175866.408981 trunc from 0x9f7af2 to 0x52f37f (0x4c8772 bytes) 000150[0] 1789175866.602434 write 0x196000 thru 0x199ed6 (0x3ed7 bytes) 000151[1] 1789175866.685810 read_OD 0x1a0000 thru 0x1a7fff (0x8000 bytes) 000152[1] 1789175866.768962 trunc from 0x52f380 to 0x5054c2 (0x29ebd bytes) 000153[0] 1789175866.897632 mapwrite 0x0dd000 thru 0x0e89fc (0xb9fd bytes) 000154[1] 1789175867.105708 read_OD 0x4cc000 thru 0x4cffff (0x4000 bytes) 000155[0] 1789175867.199512 trunc from 0x5054c3 to 0x213ca9 (0x2f1819 bytes) 000157[0] 1789175867.347918 mapread 0x124000 thru 0x12d3e2 (0x93e3 bytes) 000159[0] 1789175867.414286 read 0x028000 thru 0x0292f9 (0x12fa bytes) 000160[0] 1789175867.494522 mapwrite 0x57a000 thru 0x57fd7d (0x5d7e bytes) 000161[0] 1789175867.739606 mapwrite 0x7f4000 thru 0x7f78bc (0x38bd bytes) 000164[0] 1789175867.969900 trunc from 0x7f78bd to 0x9651a4 (0x16d8e8 bytes) 000165[0] 1789175868.008817 mapread 0x118000 thru 0x11bc69 (0x3c6a bytes) 000167[1] 1789175868.013626 mapread 0x8e6000 thru 0x8ee215 (0x8216 bytes) 000171[1] 1789175868.163357 mapwrite 0x479000 thru 0x48335e (0xa35f bytes) 000172[0] 1789175868.375541 trunc from 0x9651a5 to 0x3ef4f0 (0x575cb4 bytes) 000175[0] 1789175868.452226 punch from 0x09cc3c to 0x0aa915 (0xdcda bytes) 000176[0] 1789175868.511951 mapwrite 0x140000 thru 0x14c48c (0xc48d bytes) 000177[1] 1789175868.777766 read_OD 0x15d000 thru 0x169fff (0xd000 bytes) 000178[1] 1789175868.832758 punch from 0x2862b8 to 0x294444 (0xe18d bytes) 000180[0] 1789175868.876963 trunc from 0x3ef4f1 to 0x716bde (0x3276ee bytes) 000181[1] 1789175868.959720 punch from 0x4645fe to 0x465d1e (0x1721 bytes) 000182[0] 1789175868.990734 read 0x382000 thru 0x38f01e (0xd01f bytes) 000184[0] 1789175869.038681 write 0x0f6000 thru 0x105f70 (0xff71 bytes) 000186[1] 1789175869.106862 punch from 0x3aac3f to 0x3b486a (0x9c2c bytes) 000187[0] 1789175869.180250 mapread 0x1e0000 thru 0x1e25fe (0x25ff bytes) 000191[1] 1789175869.293622 trunc from 0x716bdf to 0x72059d (0x99bf bytes) 000192[0] 1789175869.390602 write 0x9df000 thru 0x9eee28 (0xfe29 bytes) 000193[1] 1789175869.432798 mapwrite 0x7b3000 thru 0x7b54eb (0x24ec bytes) 000194[1] 1789175869.700234 trunc from 0x9eee29 to 0x58a716 (0x464712 bytes) 000195[0] 1789175869.748595 mapread 0x377000 thru 0x3849b7 (0xd9b8 bytes) 000199[1] 1789175869.856818 punch from 0x29d922 to 0x2a4313 (0x69f2 bytes) 000201[1] 1789175869.888465 read_OD 0x336000 thru 0x336fff (0x1000 bytes) 000203[0] 1789175869.903530 punch from 0x171a3c to 0x172306 (0x8cb bytes) 000204[0] 1789175869.919568 read 0x179000 thru 0x17d564 (0x4565 bytes) 000205[0] 1789175869.929520 mapwrite 0x396000 thru 0x39df87 (0x7f88 bytes) 000206[1] 1789175870.019922 mapread 0x512000 thru 0x51a69d (0x869e bytes) 000207[0] 1789175870.061419 mapread 0x0a8000 thru 0x0a9c23 (0x1c24 bytes) 000208[0] 1789175870.074885 mapread 0x098000 thru 0x0a4fd0 (0xcfd1 bytes) 000209[0] 1789175870.099050 read 0x513000 thru 0x5186c9 (0x56ca bytes) 000211[0] 1789175870.112788 read 0x29b000 thru 0x2a74b5 (0xc4b6 bytes) 000212[0] 1789175870.124209 punch from 0x25ecba to 0x26732b (0x8672 bytes) 000213[0] 1789175870.159849 mapwrite 0x032000 thru 0x041592 (0xf593 bytes) 000215[0] 1789175870.265821 mapread 0x084000 thru 0x08a41c (0x641d bytes) 000217[1] 1789175870.343178 punch from 0x066001 to 0x069ad1 (0x3ad1 bytes) 000222[1] 1789175870.385516 read_OD 0x1e1000 thru 0x1e2fff (0x2000 bytes) 000223[0] 1789175870.407613 trunc from 0x58a717 to 0x3211ee (0x269528 bytes) 000227[1] 1789175870.561456 trunc from 0x3211ef to 0x4047bb (0xe35cd bytes) 000229[1] 1789175870.675278 write_OD 0x0cf000 thru 0x0d3fff (0x5000 bytes) 000230[0] 1789175870.843584 punch from 0x052faf to 0x062d14 (0xfd66 bytes) 000231[0] 1789175870.866785 punch from 0x040b83 to 0x043da0 (0x321e bytes) 000232[1] 1789175870.903478 mapread 0x3eb000 thru 0x3f0bdc (0x5bdd bytes) 000233[0] 1789175871.011573 read 0x352000 thru 0x35afed (0x8fee bytes) 000235[1] 1789175871.047729 mapwrite 0x2a0000 thru 0x2a7b73 (0x7b74 bytes) 000236[1] 1789175871.232800 read_OD 0x08e000 thru 0x090fff (0x3000 bytes) 000238[0] 1789175871.262739 write 0x6b7000 thru 0x6bafcf (0x3fd0 bytes) 000239[1] 1789175871.350372 read_OD 0x634000 thru 0x63afff (0x7000 bytes) 000241[0] 1789175871.408702 read 0x657000 thru 0x65afbb (0x3fbc bytes) 000242[1] 1789175871.416154 write_OD 0x779000 thru 0x77dfff (0x5000 bytes) 000245[0] 1789175871.482364 read 0x650000 thru 0x65ed69 (0xed6a bytes) 000247[1] 1789175871.493791 read_OD 0x4a2000 thru 0x4a8fff (0x7000 bytes) 000248[1] 1789175871.502207 mapwrite 0x871000 thru 0x872849 (0x184a bytes) 000249[0] 1789175871.623682 mapwrite 0x8f0000 thru 0x8f7c0a (0x7c0b bytes) 000252[0] 1789175871.752573 trunc from 0x8f7c0b to 0x943f75 (0x4c36b bytes) 000253[1] 1789175871.773618 read_OD 0x5a6000 thru 0x5affff (0xa000 bytes) 000255[1] 1789175871.811763 write_OD 0x409000 thru 0x40cfff (0x4000 bytes) 000256[1] 1789175871.886410 punch from 0x3e0cc5 to 0x3ef259 (0xe595 bytes) 000258[0] 1789175871.935972 write 0x882000 thru 0x88d167 (0xb168 bytes) 000259[0] 1789175871.962820 trunc from 0x943f76 to 0x004f25 (0x93f050 bytes) 000263[1] 1789175872.026737 trunc from 0x004f26 to 0x64ca65 (0x647b40 bytes) 000266[1] 1789175872.111555 read_OD 0x32b000 thru 0x32bfff (0x1000 bytes) 000267[0] 1789175872.130422 write 0x71f000 thru 0x72af1d (0xbf1e bytes) 000269[1] 1789175872.161497 mapwrite 0x3ce000 thru 0x3cf0ec (0x10ed bytes) 000271[0] 1789175872.280841 mapread 0x4f4000 thru 0x4fc204 (0x8205 bytes) 000274[1] 1789175872.342586 write_OD 0x86d000 thru 0x86efff (0x2000 bytes) 000276[0] 1789175872.439531 trunc from 0x86f000 to 0x3db069 (0x493f96 bytes) 000277[1] 1789175872.501699 punch from 0x1c9ab8 to 0x1c9f49 (0x492 bytes) 000278[0] 1789175872.541158 punch from 0x0598af to 0x05a838 (0xf8a bytes) 000280[1] 1789175872.592575 read_OD 0x304000 thru 0x30bfff (0x8000 bytes) 000281[1] 1789175872.624008 mapwrite 0x5ba000 thru 0x5bac8e (0xc8f bytes) 000285[0] 1789175872.837291 mapread 0x438000 thru 0x442b80 (0xab81 bytes) 000287[1] 1789175872.902772 punch from 0x0503ff to 0x05de3b (0xda3d bytes) 000291[0] 1789175872.956951 write 0x574000 thru 0x574ea5 (0xea6 bytes) 000294[0] 1789175873.020293 read 0x3d5000 thru 0x3e3ce5 (0xece6 bytes) 000295[1] 1789175873.037200 punch from 0x2fcc9d to 0x2fe67b (0x19df bytes) 000296[0] 1789175873.059402 write 0x59b000 thru 0x5a21a6 (0x71a7 bytes) 000297[0] 1789175873.085792 mapread 0x3c6000 thru 0x3d150c (0xb50d bytes) 000300[1] 1789175873.176839 mapwrite 0x566000 thru 0x574118 (0xe119 bytes) 000301[0] 1789175873.474826 trunc from 0x5bac8f to 0x3e3683 (0x1d760b bytes) 000302[0] 1789175873.523669 trunc from 0x3e3684 to 0x8db3d8 (0x4f7d55 bytes) 000305[0] 1789175873.593755 punch from 0x37a8e6 to 0x382625 (0x7d40 bytes) 000306[0] 1789175873.605943 read 0x37e000 thru 0x3876da (0x96db bytes) 000310[1] 1789175873.674810 write_OD 0x3b8000 thru 0x3bffff (0x8000 bytes) 000311[1] 1789175873.754036 mapread 0x146000 thru 0x14c930 (0x6931 bytes) 000312[0] 1789175873.803945 read 0x8d0000 thru 0x8d1be9 (0x1bea bytes) 000313[0] 1789175873.810892 punch from 0x3dc27a to 0x3dd756 (0x14dd bytes) 000314[1] 1789175873.850820 write_OD 0x8cb000 thru 0x8d3fff (0x9000 bytes) 000316[0] 1789175873.922919 read 0x4ff000 thru 0x504db5 (0x5db6 bytes) 000318[1] 1789175873.938080 read_OD 0x47b000 thru 0x489fff (0xf000 bytes) 000324[0] 1789175873.966968 write 0x29e000 thru 0x2aa64b (0xc64c bytes) 000329[0] 1789175874.091723 trunc from 0x8db3d9 to 0x199753 (0x741c85 bytes) 000331[1] 1789175874.135363 trunc from 0x199754 to 0x890681 (0x6f6f2e bytes) 000334[0] 1789175874.237705 trunc from 0x890682 to 0x0264c5 (0x86a1bc bytes) 000338[0] 1789175874.335960 mapread 0x015000 thru 0x017121 (0x2122 bytes) 000339[1] 1789175874.365018 read_OD 0x025000 thru 0x025fff (0x1000 bytes) 000340[1] 1789175874.394418 mapwrite 0x065000 thru 0x069755 (0x4756 bytes) 000341[1] 1789175874.513692 punch from 0x04636b to 0x04910f (0x2da5 bytes) 000342[1] 1789175874.519640 write_OD 0x95e000 thru 0x964fff (0x7000 bytes) 000345[1] 1789175874.587523 punch from 0x71faee to 0x72129c (0x17af bytes) 000346[0] 1789175874.604586 trunc from 0x965000 to 0x064f00 (0x9000ff bytes) 000348[0] 1789175874.668752 punch from 0x01c522 to 0x01ebb1 (0x2690 bytes) 000349[1] 1789175874.689833 mapread 0x01d000 thru 0x029722 (0xc723 bytes) 000350[0] 1789175874.773930 punch from 0x0563c7 to 0x0597c1 (0x33fb bytes) 000355[1] 1789175874.843906 read_OD 0x057000 thru 0x062fff (0xc000 bytes) 000356[0] 1789175874.888075 trunc from 0x064f01 to 0x3166a6 (0x2b17a6 bytes) 000358[1] 1789175874.955207 write_OD 0x953000 thru 0x95afff (0x8000 bytes) 000359[1] 1789175875.051809 punch from 0x7625aa to 0x76cc97 (0xa6ee bytes) 000360[1] 1789175875.087965 mapwrite 0x572000 thru 0x57ba03 (0x9a04 bytes) 000362[1] 1789175875.370972 mapwrite 0x529000 thru 0x52df24 (0x4f25 bytes) 000364[0] 1789175875.477787 read 0x429000 thru 0x438c6e (0xfc6f bytes) 000366[1] 1789175875.559293 mapwrite 0x37b000 thru 0x37ef25 (0x3f26 bytes) 000367[0] 1789175875.734815 read 0x5b1000 thru 0x5b57c3 (0x47c4 bytes) 000369[0] 1789175875.767885 mapread 0x064000 thru 0x06f9ca (0xb9cb bytes) 000370[0] 1789175875.835041 trunc from 0x95b000 to 0x1c1e29 (0x7991d6 bytes) 000371[1] 1789175875.885537 mapwrite 0x5da000 thru 0x5e7aa0 (0xdaa1 bytes) 000373[1] 1789175876.182460 trunc from 0x5e7aa1 to 0x6a9bf2 (0xc2152 bytes) 000374[0] 1789175876.243531 mapread 0x3e4000 thru 0x3e812c (0x412d bytes) 000377[1] 1789175876.345065 read_OD 0x3ba000 thru 0x3c8fff (0xf000 bytes) 000382[0] 1789175876.359471 write 0x790000 thru 0x7978be (0x78bf bytes) 000383[1] 1789175876.412669 read_OD 0x44f000 thru 0x45afff (0xc000 bytes) 000384[0] 1789175876.469651 read 0x129000 thru 0x13542c (0xc42d bytes) 000385[0] 1789175876.490658 trunc from 0x7978bf to 0x28430e (0x5135b0 bytes) 000386[1] 1789175876.534593 mapread 0x07d000 thru 0x0803a3 (0x33a4 bytes) 000389[0] 1789175876.603345 write 0x7dc000 thru 0x7ebc56 (0xfc57 bytes) 000390[1] 1789175876.630593 trunc from 0x7ebc57 to 0x645f70 (0x1a5ce6 bytes) 000392[0] 1789175876.698319 trunc from 0x645f71 to 0x939d3f (0x2f3dcf bytes) 000393[1] 1789175876.747511 mapwrite 0x708000 thru 0x70b89d (0x389e bytes) 000398[1] 1789175877.005629 punch from 0x1ebbd0 to 0x1f349a (0x78cb bytes) 000400[1] 1789175877.057360 mapwrite 0x793000 thru 0x796321 (0x3322 bytes) 000401[0] 1789175877.297312 mapwrite 0x114000 thru 0x1170b4 (0x30b5 bytes) 000402[0] 1789175877.560671 punch from 0x7b422b to 0x7bbe81 (0x7c57 bytes) 000404[1] 1789175877.648602 mapread 0x7ce000 thru 0x7d96a3 (0xb6a4 bytes) 000405[1] 1789175877.765525 mapwrite 0x31d000 thru 0x32c654 (0xf655 bytes) 000406[1] 1789175877.954882 mapread 0x08f000 thru 0x09de5b (0xee5c bytes) 000407[0] 1789175878.069734 read 0x200000 thru 0x208ae7 (0x8ae8 bytes) 000408[1] 1789175878.109477 mapread 0x2f8000 thru 0x300f48 (0x8f49 bytes) 000411[0] 1789175878.139930 punch from 0x92e012 to 0x937e86 (0x9e75 bytes) 000413[0] 1789175878.159799 trunc from 0x939d40 to 0x53a1b2 (0x3ffb8d bytes) 000414[0] 1789175878.202403 trunc from 0x53a1b3 to 0x68e29b (0x1540e9 bytes) 000416[0] 1789175878.261395 trunc from 0x68e29c to 0x65d484 (0x30e17 bytes) 000418[1] 1789175878.306434 write_OD 0x061000 thru 0x06ffff (0xf000 bytes) 000419[0] 1789175878.376405 write 0x2ec000 thru 0x2f5f3c (0x9f3d bytes) 000420[0] 1789175878.398733 mapread 0x5de000 thru 0x5e058e (0x258f bytes) 000422[0] 1789175878.434978 trunc from 0x65d485 to 0x64007f (0x1d405 bytes) 000426[1] 1789175878.516379 write_OD 0x525000 thru 0x52cfff (0x8000 bytes) 000427[1] 1789175878.588514 mapread 0x3cc000 thru 0x3d0d2b (0x4d2c bytes) 000428[0] 1789175878.602609 read 0x0e3000 thru 0x0f22f8 (0xf2f9 bytes) 000433[0] 1789175878.620843 trunc from 0x640080 to 0x6afd78 (0x6fcf9 bytes) 000434[1] 1789175878.673378 write_OD 0x869000 thru 0x871fff (0x9000 bytes) 000437[0] 1789175878.736543 trunc from 0x872000 to 0x2e951c (0x588ae3 bytes) 000438[1] 1789175878.782675 mapwrite 0x5ec000 thru 0x5f16a8 (0x56a9 bytes) 000439[0] 1789175878.955014 mapwrite 0x8f3000 thru 0x900414 (0xd415 bytes) 000441[1] 1789175879.099436 read_OD 0x5d7000 thru 0x5dbfff (0x5000 bytes) 000442[0] 1789175879.178629 mapwrite 0x07b000 thru 0x07dbce (0x2bcf bytes) 000444[1] 1789175879.361113 write_OD 0x40f000 thru 0x412fff (0x4000 bytes) 000445[1] 1789175879.466973 read_OD 0x5d9000 thru 0x5e2fff (0xa000 bytes) 000449[1] 1789175879.516004 punch from 0x28bd1d to 0x28fa4c (0x3d30 bytes) 000452[1] 1789175879.621283 write_OD 0x1fb000 thru 0x206fff (0xc000 bytes) 000454[1] 1789175879.723767 mapread 0x4d3000 thru 0x4dc836 (0x9837 bytes) 000456[1] 1789175879.760260 write_OD 0x994000 thru 0x99dfff (0xa000 bytes) 000457[1] 1789175879.899529 read_OD 0x1c6000 thru 0x1cffff (0xa000 bytes) 000459[1] 1789175879.919662 mapread 0x204000 thru 0x20e0be (0xa0bf bytes) 000461[0] 1789175879.960930 trunc from 0x99e000 to 0x1660ce (0x837f31 bytes) 000463[0] 1789175880.126765 write 0x072000 thru 0x07b7e2 (0x97e3 bytes) 000464[1] 1789175880.198781 mapwrite 0x82b000 thru 0x8397fc (0xe7fd bytes) 000465[1] 1789175880.445553 write_OD 0x7d1000 thru 0x7ddfff (0xd000 bytes) 000469[0] 1789175880.642275 mapread 0x17c000 thru 0x18bb37 (0xfb38 bytes) 000470[1] 1789175880.708624 mapread 0x610000 thru 0x61bfdb (0xbfdc bytes) 000471[0] 1789175880.805262 write 0x2f3000 thru 0x2f7cfb (0x4cfc bytes) 000472[1] 1789175880.871055 mapread 0x280000 thru 0x28ab5f (0xab60 bytes) 000473[0] 1789175881.040941 mapwrite 0x1b2000 thru 0x1b2d64 (0xd65 bytes) 000474[0] 1789175881.303761 mapread 0x345000 thru 0x34ff1f (0xaf20 bytes) 000476[0] 1789175881.418490 trunc from 0x8397fd to 0x95e984 (0x125188 bytes) 000479[0] 1789175881.544912 mapread 0x367000 thru 0x3744e0 (0xd4e1 bytes) 000482[0] 1789175881.632436 trunc from 0x95e985 to 0x30734a (0x65763a bytes) 000484[1] 1789175881.721021 punch from 0x18a3d5 to 0x1952e5 (0xaf11 bytes) 000486[0] 1789175881.798329 read 0x0e4000 thru 0x0eddbc (0x9dbd bytes) 000490[1] 1789175881.966716 punch from 0x229024 to 0x22e93e (0x591b bytes) 000492[0] 1789175882.022957 punch from 0x005af1 to 0x0091b7 (0x36c7 bytes) 000493[0] 1789175882.058784 read 0x267000 thru 0x26e6eb (0x76ec bytes) 000495[0] 1789175882.163952 write 0x3e0000 thru 0x3e49e3 (0x49e4 bytes) 000496[1] 1789175882.210126 trunc from 0x3e49e4 to 0x0a89f9 (0x33bfea bytes) 000498[1] 1789175882.390290 write_OD 0x424000 thru 0x42bfff (0x8000 bytes) 000499[0] 1789175882.569895 mapread 0x3da000 thru 0x3da815 (0x816 bytes) 000500[1] 1789175882.666666 write_OD 0x80a000 thru 0x816fff (0xd000 bytes) All operations completed A-OK! PASS 16a (70s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:18:09 (1789175889) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.7135 s, 11.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.484355 s, 17.3 MB/s Chance of close/open is 1 in 50 Seed set to 5893 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero length punch hole truncating to largest ever: 0x482f51 000003[1] 1789175894.551148 trunc from 00000000 to 0x482f50 (0x482f51 bytes) 000004[1] 1789175894.603126 mapread 0x2cdbb2 thru 0x2cf5f3 (0x1a42 bytes) 000005[1] 1789175894.661145 read 0x233d00 thru 0x234276 (0x577 bytes) truncating to largest ever: 0x5af75a 000006[1] 1789175894.708611 trunc from 0x482f51 to 0x5af759 (0x12c809 bytes) 000007[1] 1789175894.779713 mapread 0x1af8a9 thru 0x1afd97 (0x4ef bytes) 000008[1] 1789175894.845166 punch from 0x0ed3bb to 0x0eef91 (0x1bd7 bytes) 000009[1] 1789175894.886558 mapwrite 0x39688a thru 0x3975cf (0xd46 bytes) 000010[0] 1789175894.975000 write 0x205478 thru 0x205dee (0x977 bytes) 000011[1] 1789175895.007941 mapread 0x2e9eda thru 0x2ebc3d (0x1d64 bytes) 000015[0] 1789175895.083482 trunc from 0x5af75a to 0x1f3929 (0x3bbe30 bytes) 000016[1] 1789175895.170499 trunc from 0x1f392a to 0x1311a3 (0xc2786 bytes) 000018[0] 1789175895.257358 read 0x0f5d70 thru 0x0f73ab (0x163c bytes) 000022[1] 1789175895.341230 trunc from 0x1311a4 to 0x299802 (0x16865f bytes) 000025[1] 1789175895.471295 write 0x2f053c thru 0x2f15f8 (0x10bd bytes) 000027[1] 1789175895.526226 mapread 0x28119e thru 0x282da7 (0x1c0a bytes) 000028[1] 1789175895.544273 read 0x2968cf thru 0x2977b7 (0xee9 bytes) 000031[1] 1789175895.616679 mapread 0x2342da thru 0x2349f3 (0x71a bytes) 000032[0] 1789175895.638968 mapread 0x03efff thru 0x03ff1c (0xf1e bytes) 000033[1] 1789175895.706236 mapread 0x08974b thru 0x08aef9 (0x17af bytes) 000035[0] 1789175895.720709 write 0x2dacd3 thru 0x2dc54f (0x187d bytes) 000040[0] 1789175895.831575 read 0x1ab9c5 thru 0x1ac294 (0x8d0 bytes) 000043[1] 1789175895.949354 mapread 0x2eaf5e thru 0x2ebc38 (0xcdb bytes) 000044[0] 1789175896.016900 mapread 0x2eec58 thru 0x2ef3d2 (0x77b bytes) 000045[0] 1789175896.044382 mapread 0x078c29 thru 0x07994e (0xd26 bytes) truncating to largest ever: 0x68f717 000046[1] 1789175896.079497 trunc from 0x2f15f9 to 0x68f716 (0x39e11e bytes) 000047[0] 1789175896.150390 mapread 0x5b4235 thru 0x5b4e63 (0xc2f bytes) 000048[1] 1789175896.197849 punch from 0x4866c6 to 0x486b52 (0x48d bytes) 000050[1] 1789175896.247695 trunc from 0x68f717 to 0x46214a (0x22d5cc bytes) 000051[1] 1789175896.297144 write 0x3d00c1 thru 0x3d0d2e (0xc6e bytes) 000052[0] 1789175896.347398 mapread 0x22e092 thru 0x22e852 (0x7c1 bytes) 000054[1] 1789175896.437895 mapwrite 0x339418 thru 0x33a1ca (0xdb3 bytes) 000056[1] 1789175896.537577 mapread 0x017e5f thru 0x018b6a (0xd0c bytes) 000057[0] 1789175896.550339 read 0x196ace thru 0x19826e (0x17a1 bytes) 000058[0] 1789175896.586013 read 0x14af96 thru 0x14b5ae (0x619 bytes) 000062[0] 1789175896.631348 write 0x659fc6 thru 0x65b503 (0x153e bytes) 000063[1] 1789175896.647308 write 0x824da4 thru 0x8258b0 (0xb0d bytes) 000065[1] 1789175896.694976 read 0x654382 thru 0x655518 (0x1197 bytes) 000067[1] 1789175896.759056 write 0x19bc03 thru 0x19c0bb (0x4b9 bytes) 000072[0] 1789175896.798280 write 0x105ebd thru 0x107619 (0x175d bytes) 000073[0] 1789175896.828574 read 0x7fb7f5 thru 0x7fbd3c (0x548 bytes) 000074[0] 1789175896.834028 write 0x3a9936 thru 0x3aafd2 (0x169d bytes) 000075[1] 1789175896.843041 write 0x480da5 thru 0x480e50 (0xac bytes) 000076[0] 1789175896.877052 punch from 0x34de5c to 0x34e168 (0x30d bytes) 000078[1] 1789175896.937859 mapread 0x206852 thru 0x2082e5 (0x1a94 bytes) truncating to largest ever: 0x9b7f8d 000079[1] 1789175896.989152 trunc from 0x8258b1 to 0x9b7f8c (0x1926dc bytes) 000081[1] 1789175897.141692 mapwrite 0x7cc5a1 thru 0x7cce63 (0x8c3 bytes) 000082[1] 1789175897.251840 trunc from 0x9b7f8d to 0x1bff25 (0x7f8067 bytes) 000085[1] 1789175897.440289 mapwrite 0x11bb5b thru 0x11d98a (0x1e30 bytes) 000087[0] 1789175897.650376 read 0x0c53d5 thru 0x0c5bdb (0x807 bytes) 000088[1] 1789175897.804857 read 0x0980be thru 0x099402 (0x1345 bytes) 000093[1] 1789175897.934428 mapwrite 0x3890b6 thru 0x389c73 (0xbbe bytes) 000094[1] 1789175898.118125 write 0x729c71 thru 0x72b97d (0x1d0d bytes) 000095[0] 1789175898.123822 read 0x67ad64 thru 0x67cc60 (0x1efd bytes) 000096[1] 1789175898.184684 punch from 0x4e1064 to 0x4e1627 (0x5c4 bytes) 000098[1] 1789175898.207679 mapread 0x4ae9f5 thru 0x4b092d (0x1f39 bytes) 000099[0] 1789175898.225758 punch from 0x44b8a0 to 0x44c47e (0xbdf bytes) 000100[0] 1789175898.257354 mapwrite 0x0d2fc4 thru 0x0d3d19 (0xd56 bytes) 000101[1] 1789175898.508849 mapread 0x3de350 thru 0x3df0d9 (0xd8a bytes) 000102[1] 1789175898.704415 read 0x53cd6e thru 0x53ecb4 (0x1f47 bytes) 000105[0] 1789175898.744991 write 0x99765c thru 0x9976bf (0x64 bytes) 000106[1] 1789175898.822619 punch from 0x6931b5 to 0x6943fd (0x1249 bytes) 000107[1] 1789175898.891234 trunc from 0x9976c0 to 0x568da9 (0x42e916 bytes) 000108[1] 1789175898.987481 punch from 0x17e7ca to 0x17ff71 (0x17a8 bytes) 000109[0] 1789175899.025659 trunc from 0x568daa to 0x698317 (0x12f56e bytes) 000110[0] 1789175899.086146 read 0x4cc1e7 thru 0x4cc3f2 (0x20c bytes) 000111[1] 1789175899.124060 trunc from 0x698318 to 0x03c5d2 (0x65bd45 bytes) 000118[1] 1789175899.285593 mapwrite 0x30c790 thru 0x30d137 (0x9a8 bytes) 000119[0] 1789175899.366994 trunc from 0x30d138 to 0x1d6c97 (0x1364a0 bytes) 000120[1] 1789175899.410825 read 0x11b91a thru 0x11ca3d (0x1124 bytes) 000122[1] 1789175899.516675 write 0x5ff155 thru 0x5ffdf3 (0xc9f bytes) 000125[1] 1789175899.569541 write 0x640f2b thru 0x641112 (0x1e8 bytes) 000126[0] 1789175899.570839 mapread 0x51d49f thru 0x51e0ef (0xc51 bytes) 000127[0] 1789175899.615953 trunc from 0x641113 to 0x593dd5 (0xad33d bytes) 000128[0] 1789175899.679946 read 0x52e7b8 thru 0x52f296 (0xadf bytes) 000129[0] 1789175899.687187 trunc from 0x593dd6 to 0x3b658a (0x1dd84b bytes) 000130[1] 1789175899.737832 punch from 0x1733da to 0x174361 (0xf88 bytes) 000131[0] 1789175899.778308 read 0x313dd9 thru 0x315b81 (0x1da9 bytes) 000133[1] 1789175899.888033 write 0x1a766b thru 0x1a8a29 (0x13bf bytes) 000134[1] 1789175899.984647 punch from 0x0ce5b7 to 0x0cec9c (0x6e6 bytes) 000138[1] 1789175900.052787 mapread 0x15eedd thru 0x160610 (0x1734 bytes) 000139[1] 1789175900.093808 mapread 0x198ce4 thru 0x19a411 (0x172e bytes) 000141[1] 1789175900.153938 mapread 0x0ee55d thru 0x0ef545 (0xfe9 bytes) 000144[0] 1789175900.278472 mapwrite 0x1d09c3 thru 0x1d1f1a (0x1558 bytes) 000145[1] 1789175900.449811 mapwrite 0x59d092 thru 0x59ec0b (0x1b7a bytes) 000149[1] 1789175900.708468 punch from 0x15ebbd to 0x15fd56 (0x119a bytes) 000151[0] 1789175900.716188 write 0x7951e2 thru 0x7969d7 (0x17f6 bytes) 000152[1] 1789175900.756776 read 0x2b82bc thru 0x2b8f1e (0xc63 bytes) 000153[0] 1789175900.786039 mapread 0x393a64 thru 0x39425f (0x7fc bytes) 000154[1] 1789175900.860036 read 0x329a5d thru 0x32b4aa (0x1a4e bytes) 000156[1] 1789175900.993741 write 0x1bb75a thru 0x1bb7ab (0x52 bytes) 000157[0] 1789175901.072705 write 0x049b19 thru 0x04a278 (0x760 bytes) 000158[0] 1789175901.163532 trunc from 0x7969d8 to 0x814533 (0x7db5c bytes) 000159[0] 1789175901.208571 write 0x9037c5 thru 0x905028 (0x1864 bytes) 000162[1] 1789175901.245808 write 0x5d00d3 thru 0x5d082b (0x759 bytes) 000164[1] 1789175901.439003 mapread 0x007eaf thru 0x009d41 (0x1e93 bytes) 000165[0] 1789175901.541266 write 0x742f60 thru 0x744740 (0x17e1 bytes) 000168[1] 1789175901.812509 mapread 0x5bb321 thru 0x5bca04 (0x16e4 bytes) 000170[1] 1789175901.871047 mapread 0x5e1e63 thru 0x5e3108 (0x12a6 bytes) 000172[0] 1789175901.914606 punch from 0x339d91 to 0x33b5da (0x184a bytes) 000173[0] 1789175901.972821 punch from 0x2f37f0 to 0x2f4c80 (0x1491 bytes) 000175[1] 1789175902.004537 read 0x45365c thru 0x4536e0 (0x85 bytes) 000177[1] 1789175902.032230 trunc from 0x905029 to 0x515034 (0x3efff4 bytes) 000178[1] 1789175902.115070 punch from 0x20f92f to 0x210c02 (0x12d4 bytes) 000180[0] 1789175902.166552 punch from 0x3c44e0 to 0x3c520d (0xd2e bytes) 000181[0] 1789175902.253761 mapwrite 0x76374a thru 0x763e13 (0x6ca bytes) 000184[1] 1789175902.447748 read 0x663941 thru 0x6641b4 (0x874 bytes) 000185[0] 1789175902.497749 punch from 0x2b3d42 to 0x2b4189 (0x448 bytes) 000188[1] 1789175902.561970 mapwrite 0x56fb21 thru 0x5705e7 (0xac7 bytes) 000189[1] 1789175902.712890 trunc from 0x763e14 to 0x3e4eb0 (0x37ef63 bytes) 000191[1] 1789175902.891798 mapread 0x38b66b thru 0x38c382 (0xd18 bytes) 000192[1] 1789175902.935077 read 0x3d9983 thru 0x3db928 (0x1fa6 bytes) 000194[0] 1789175903.098602 punch from 0x05ec85 to 0x05eebf (0x23b bytes) 000195[1] 1789175903.149169 mapwrite 0x36a1c5 thru 0x36bd9a (0x1bd6 bytes) 000197[1] 1789175903.270861 punch from 0x37bd05 to 0x37c7a5 (0xaa1 bytes) 000198[0] 1789175903.287279 mapread 0x0d8341 thru 0x0da2a2 (0x1f62 bytes) 000199[0] 1789175903.371479 read 0x1b7da5 thru 0x1b9531 (0x178d bytes) 000201[0] 1789175903.522321 write 0x39020a thru 0x391cc5 (0x1abc bytes) 000202[1] 1789175903.609315 mapwrite 0x8632f6 thru 0x8652c4 (0x1fcf bytes) 000203[1] 1789175903.988583 mapwrite 0x39de2b thru 0x39f862 (0x1a38 bytes) 000204[0] 1789175904.149945 read 0x24db20 thru 0x24e571 (0xa52 bytes) 000205[0] 1789175904.188770 punch from 0x739194 to 0x739a01 (0x86e bytes) 000207[0] 1789175904.276827 mapwrite 0x05c9b9 thru 0x05d763 (0xdab bytes) 000209[0] 1789175904.489647 punch from 0x267233 to 0x267854 (0x622 bytes) 000210[0] 1789175904.552783 punch from 0x65af83 to 0x65ca2b (0x1aa9 bytes) 000211[0] 1789175904.585810 mapread 0x42fe92 thru 0x4309c6 (0xb35 bytes) 000213[0] 1789175904.626607 write 0x4476f3 thru 0x448345 (0xc53 bytes) 000214[1] 1789175904.690026 mapwrite 0x14953c thru 0x14ab1b (0x15e0 bytes) 000215[0] 1789175904.871914 mapwrite 0x64cd3b thru 0x64d82b (0xaf1 bytes) 000217[1] 1789175905.134702 mapread 0x1065b7 thru 0x10766c (0x10b6 bytes) 000218[0] 1789175905.180691 trunc from 0x8652c5 to 0x5ca6db (0x29abe9 bytes) 000220[0] 1789175905.255929 trunc from 0x5ca6dc to 0x3e8330 (0x1e23ab bytes) 000221[0] 1789175905.327550 punch from 0x1a3f8d to 0x1a5dec (0x1e60 bytes) 000222[0] 1789175905.359215 read 0x352486 thru 0x3524a2 (0x1d bytes) 000223[0] 1789175905.448704 trunc from 0x3e8331 to 0x76faea (0x3877ba bytes) 000224[1] 1789175905.476874 write 0x9df175 thru 0x9e08f3 (0x177f bytes) 000225[0] 1789175905.513855 read 0x3353bf thru 0x335f17 (0xb59 bytes) 000227[1] 1789175905.557231 write 0x23339a thru 0x235015 (0x1c7c bytes) 000228[1] 1789175905.601551 write 0x595df2 thru 0x5964f9 (0x708 bytes) 000230[0] 1789175905.651834 write 0x873514 thru 0x87363d (0x12a bytes) 000231[0] 1789175905.701565 mapwrite 0x50dfe9 thru 0x50fcb0 (0x1cc8 bytes) 000232[0] 1789175905.861583 mapwrite 0x9c2dc3 thru 0x9c3580 (0x7be bytes) 000233[1] 1789175905.926310 punch from 0x4b16b8 to 0x4b2ba0 (0x14e9 bytes) 000234[1] 1789175905.963533 trunc from 0x9e08f4 to 0x1b071c (0x8301d7 bytes) 000235[1] 1789175906.009192 trunc from 0x1b071d to 0x7f90d9 (0x6489bd bytes) 000237[1] 1789175906.087748 mapread 0x67f53c thru 0x67fa1f (0x4e4 bytes) 000238[1] 1789175906.093915 mapread 0x3527a1 thru 0x3538ff (0x115f bytes) 000239[1] 1789175906.166859 write 0x06b682 thru 0x06baf4 (0x473 bytes) 000240[1] 1789175906.207490 read 0x3eee39 thru 0x3f0ce1 (0x1ea9 bytes) 000242[1] 1789175906.256258 write 0x3674b2 thru 0x368ce6 (0x1835 bytes) 000243[0] 1789175906.270712 read 0x6f63ec thru 0x6f8368 (0x1f7d bytes) 000247[1] 1789175906.418985 write 0x419cac thru 0x41a4bf (0x814 bytes) 000249[1] 1789175906.573920 mapread 0x22da1a thru 0x22dc5a (0x241 bytes) 000251[1] 1789175906.606788 mapwrite 0x097ae6 thru 0x09815b (0x676 bytes) 000252[0] 1789175906.688703 trunc from 0x7f90da to 0x00fe54 (0x7e9285 bytes) 000253[0] 1789175906.792982 write 0x0e14a8 thru 0x0e1fe8 (0xb41 bytes) 000254[0] 1789175906.801759 read 0x08971f thru 0x0898a0 (0x182 bytes) 000257[0] 1789175906.898144 punch from 0x06f09f to 0x06f5df (0x541 bytes) 000258[0] 1789175906.949889 mapwrite 0x80cf1b thru 0x80d234 (0x31a bytes) 000260[1] 1789175907.315657 trunc from 0x80d235 to 0x028c2f (0x7e4605 bytes) 000261[1] 1789175907.386734 write 0x5a823f thru 0x5a9b8d (0x194f bytes) 000262[0] 1789175907.424868 mapread 0x1ec0ee thru 0x1eca2e (0x941 bytes) 000263[0] 1789175907.509982 write 0x5005c4 thru 0x5020f2 (0x1b2f bytes) 000264[1] 1789175907.573964 mapwrite 0x3628c0 thru 0x3638a0 (0xfe1 bytes) 000265[0] 1789175907.757961 mapwrite 0x90a2c4 thru 0x90b760 (0x149d bytes) 000267[0] 1789175907.961126 mapwrite 0x9caee1 thru 0x9ccc1e (0x1d3e bytes) 000268[0] 1789175908.050431 punch from 0x083715 to 0x084643 (0xf2f bytes) 000269[1] 1789175908.082388 punch from 0x9bdffa to 0x9be7d2 (0x7d9 bytes) 000270[0] 1789175908.136239 read 0x72831e thru 0x728927 (0x60a bytes) 000271[0] 1789175908.168194 write 0x372aff thru 0x37352c (0xa2e bytes) 000272[1] 1789175908.258445 write 0x10515d thru 0x106b22 (0x19c6 bytes) 000273[0] 1789175908.299879 punch from 0x6a79ea to 0x6a7c20 (0x237 bytes) 000275[1] 1789175908.342934 trunc from 0x9ccc1f to 0x1009c3 (0x8cc25b bytes) 000276[0] 1789175908.414487 read 0x0fc427 thru 0x0fd286 (0xe60 bytes) 000277[0] 1789175908.481413 punch from 0x064015 to 0x06538a (0x1376 bytes) 000281[0] 1789175908.572366 trunc from 0x1009c4 to 0x75656a (0x655ba7 bytes) 000282[0] 1789175908.691630 read 0x0f3452 thru 0x0f3a82 (0x631 bytes) 000284[0] 1789175908.758962 read 0x390851 thru 0x390900 (0xb0 bytes) 000285[1] 1789175908.804972 trunc from 0x75656b to 0x77875d (0x221f3 bytes) 000287[0] 1789175908.894182 write 0x124d1c thru 0x125ff0 (0x12d5 bytes) 000289[1] 1789175908.984682 mapwrite 0x9ada91 thru 0x9af14b (0x16bb bytes) 000290[0] 1789175909.187299 write 0x573b23 thru 0x5748ab (0xd89 bytes) 000293[1] 1789175909.426762 mapread 0x28d50e thru 0x28eccc (0x17bf bytes) 000294[1] 1789175909.490552 mapread 0x4487e2 thru 0x449771 (0xf90 bytes) 000297[1] 1789175909.588160 punch from 0x065a22 to 0x0670a8 (0x1687 bytes) 000299[0] 1789175909.627688 read 0x08e825 thru 0x08f091 (0x86d bytes) 000301[1] 1789175909.653149 punch from 0x604ab6 to 0x6060b9 (0x1604 bytes) 000302[1] 1789175909.678249 mapread 0x916fa6 thru 0x9171fd (0x258 bytes) 000303[0] 1789175909.691879 read 0x593273 thru 0x594d73 (0x1b01 bytes) 000306[0] 1789175909.716404 read 0x294807 thru 0x2950a3 (0x89d bytes) 000308[0] 1789175909.724169 mapwrite 0x990b2b thru 0x992108 (0x15de bytes) 000309[0] 1789175909.886151 punch from 0x55c720 to 0x55d6b2 (0xf93 bytes) 000312[1] 1789175910.037966 read 0x678aa4 thru 0x67a04c (0x15a9 bytes) 000313[0] 1789175910.096057 read 0x00e26b thru 0x00f6e6 (0x147c bytes) 000315[1] 1789175910.144017 mapread 0x322342 thru 0x3225d8 (0x297 bytes) 000316[0] 1789175910.196838 trunc from 0x9af14c to 0x5bcdae (0x3f239d bytes) 000317[1] 1789175910.413354 trunc from 0x5bcdaf to 0x59c584 (0x2082a bytes) 000318[1] 1789175910.528397 read 0x49369f thru 0x4951bb (0x1b1d bytes) 000319[0] 1789175910.619053 write 0x6a5f00 thru 0x6a643c (0x53d bytes) 000320[1] 1789175910.687995 write 0x68cc7a thru 0x68d999 (0xd20 bytes) 000322[1] 1789175910.788856 trunc from 0x6a643d to 0x381755 (0x324ce7 bytes) 000324[0] 1789175911.041446 trunc from 0x381756 to 0x543acf (0x1c237a bytes) 000325[1] 1789175911.181263 trunc from 0x543ad0 to 0x151904 (0x3f21cb bytes) 000326[1] 1789175911.237131 mapread 0x06b710 thru 0x06ba4b (0x33c bytes) 000328[0] 1789175911.297603 read 0x09be60 thru 0x09d4ab (0x164c bytes) 000330[1] 1789175911.375681 mapwrite 0x5c2ab2 thru 0x5c451c (0x1a6b bytes) 000331[0] 1789175911.476007 trunc from 0x5c451d to 0x766c52 (0x1a2736 bytes) 000332[0] 1789175911.530364 read 0x6b0f38 thru 0x6b226b (0x1334 bytes) 000334[0] 1789175911.569661 read 0x676e2d thru 0x67824f (0x1423 bytes) 000335[1] 1789175911.585539 read 0x6e9357 thru 0x6ea8ba (0x1564 bytes) 000337[1] 1789175911.679436 read 0x096b10 thru 0x09829a (0x178b bytes) 000338[0] 1789175911.704344 mapread 0x1c8233 thru 0x1c9b83 (0x1951 bytes) 000339[0] 1789175911.723411 read 0x0175b2 thru 0x017ac5 (0x514 bytes) 000340[0] 1789175911.729967 mapread 0x6d4aff thru 0x6d5b8b (0x108d bytes) 000342[0] 1789175911.740667 mapread 0x559fec thru 0x55b7a1 (0x17b6 bytes) 000343[1] 1789175911.769462 write 0x37ba58 thru 0x37c752 (0xcfb bytes) 000345[1] 1789175911.840320 mapread 0x47e09d thru 0x47f61c (0x1580 bytes) 000346[0] 1789175911.869608 read 0x0c5517 thru 0x0c636e (0xe58 bytes) 000347[0] 1789175911.915480 write 0x4e52f4 thru 0x4e6bca (0x18d7 bytes) 000349[1] 1789175911.973127 mapwrite 0x6fc81e thru 0x6fd8a9 (0x108c bytes) 000350[1] 1789175912.218651 mapread 0x4b8831 thru 0x4b8b6d (0x33d bytes) 000351[0] 1789175912.232335 mapwrite 0x52d592 thru 0x52e733 (0x11a2 bytes) 000352[1] 1789175912.447543 mapwrite 0x66389d thru 0x664552 (0xcb6 bytes) 000354[0] 1789175912.610554 trunc from 0x766c53 to 0x1d7a88 (0x58f1ca bytes) 000355[0] 1789175912.749977 mapread 0x1717a7 thru 0x172e8b (0x16e5 bytes) 000356[1] 1789175912.779783 mapread 0x121a4c thru 0x121b64 (0x119 bytes) 000357[0] 1789175912.923266 read 0x00e882 thru 0x00ec84 (0x403 bytes) 000358[1] 1789175913.016755 read 0x173ebb thru 0x175534 (0x167a bytes) 000359[1] 1789175913.072691 punch from 0x09361e to 0x09453b (0xf1e bytes) 000364[0] 1789175913.263198 read 0x16fc20 thru 0x170a43 (0xe24 bytes) 000367[1] 1789175913.380496 trunc from 0x1d7a89 to 0x515c5b (0x33e1d3 bytes) 000370[1] 1789175913.654783 mapread 0x1dc8b0 thru 0x1dd289 (0x9da bytes) 000371[0] 1789175913.818406 write 0x1c71d2 thru 0x1c89d1 (0x1800 bytes) 000372[0] 1789175914.000669 mapread 0x3d8df4 thru 0x3da17f (0x138c bytes) 000374[0] 1789175914.067288 mapwrite 0x12f902 thru 0x131555 (0x1c54 bytes) 000376[1] 1789175914.216975 mapread 0x1ac3c2 thru 0x1ad2e6 (0xf25 bytes) 000379[0] 1789175914.347301 read 0x35b227 thru 0x35c1cc (0xfa6 bytes) 000382[1] 1789175914.391450 trunc from 0x515c5c to 0x957794 (0x441b39 bytes) 000384[0] 1789175914.584045 punch from 0x356a85 to 0x357d59 (0x12d5 bytes) 000385[1] 1789175914.649731 mapwrite 0x58b9e5 thru 0x58c9e5 (0x1001 bytes) 000386[1] 1789175914.842064 mapwrite 0x837434 thru 0x837bca (0x797 bytes) 000387[1] 1789175915.120311 read 0x14c397 thru 0x14d6c7 (0x1331 bytes) 000388[0] 1789175915.178904 read 0x2d3c40 thru 0x2d430b (0x6cc bytes) 000392[0] 1789175915.259967 trunc from 0x957795 to 0x21dc2d (0x739b67 bytes) 000394[1] 1789175915.353798 read 0x09aeea thru 0x09bec1 (0xfd8 bytes) 000395[0] 1789175915.454562 mapread 0x0cef7a thru 0x0cfdaf (0xe36 bytes) 000397[0] 1789175915.483034 mapwrite 0x251b91 thru 0x252ff5 (0x1465 bytes) 000398[0] 1789175915.653586 trunc from 0x252ff6 to 0x3f6472 (0x1a347d bytes) 000399[1] 1789175915.722377 punch from 0x16dd8d to 0x16f7eb (0x1a5f bytes) 000401[1] 1789175915.801282 mapwrite 0x5f7907 thru 0x5f865b (0xd55 bytes) 000402[1] 1789175916.156457 mapwrite 0x5df396 thru 0x5e0cd9 (0x1944 bytes) 000404[1] 1789175916.328919 punch from 0x5ef582 to 0x5f0d24 (0x17a3 bytes) 000406[1] 1789175916.381961 read 0x2fd418 thru 0x2ff159 (0x1d42 bytes) 000407[1] 1789175916.383284 trunc from 0x5f865c to 0x552747 (0xa5f14 bytes) 000410[0] 1789175916.454905 mapwrite 0x1c4f38 thru 0x1c6f13 (0x1fdc bytes) 000411[1] 1789175916.684539 read 0x13f2a2 thru 0x140e54 (0x1bb3 bytes) 000412[0] 1789175916.744259 read 0x524e6f thru 0x5264be (0x1650 bytes) 000413[0] 1789175916.765729 punch from 0x242af9 to 0x243f32 (0x143a bytes) 000414[0] 1789175916.793072 mapwrite 0x2d67be thru 0x2d7a26 (0x1269 bytes) 000417[1] 1789175916.906640 punch from 0x5135a2 to 0x515291 (0x1cf0 bytes) 000419[1] 1789175916.969138 write 0x354b21 thru 0x354f18 (0x3f8 bytes) 000423[0] 1789175916.999680 punch from 0x1557fc to 0x156adf (0x12e4 bytes) 000424[0] 1789175917.059689 mapread 0x549590 thru 0x54b4ff (0x1f70 bytes) 000431[1] 1789175917.271723 mapread 0x1a1527 thru 0x1a300b (0x1ae5 bytes) 000435[0] 1789175917.328653 mapread 0x0334ba thru 0x03471c (0x1263 bytes) 000438[1] 1789175917.346586 mapread 0x18017d thru 0x181a0c (0x1890 bytes) 000440[0] 1789175917.362457 punch from 0x17a817 to 0x17bee7 (0x16d1 bytes) 000446[0] 1789175917.466441 trunc from 0x552748 to 0x4d165a (0x810ed bytes) 000449[1] 1789175917.558300 punch from 0x04815f to 0x048758 (0x5fa bytes) 000451[0] 1789175917.636737 trunc from 0x4d165b to 0x100802 (0x3d0e58 bytes) 000454[0] 1789175917.735843 read 0x05bcdd thru 0x05c738 (0xa5c bytes) 000455[1] 1789175917.854910 write 0x5893aa thru 0x58a9f1 (0x1648 bytes) 000457[1] 1789175917.956442 mapwrite 0x48a3b2 thru 0x48b275 (0xec4 bytes) 000458[1] 1789175918.072330 write 0x25049e thru 0x250c20 (0x783 bytes) 000459[0] 1789175918.110562 trunc from 0x58a9f2 to 0x69b04f (0x11065e bytes) 000460[0] 1789175918.232961 trunc from 0x69b050 to 0x12e3b9 (0x56cc96 bytes) 000462[0] 1789175918.277707 trunc from 0x12e3ba to 0x33bbe8 (0x20d82f bytes) 000465[1] 1789175918.404766 write 0x802cf0 thru 0x804a58 (0x1d69 bytes) 000466[0] 1789175918.463727 read 0x5646fe thru 0x5664c7 (0x1dca bytes) 000467[1] 1789175918.602065 punch from 0x05f6da to 0x06140e (0x1d35 bytes) 000468[0] 1789175918.633942 punch from 0x2f92f7 to 0x2f9d8e (0xa98 bytes) 000469[1] 1789175918.669308 read 0x0572f6 thru 0x057795 (0x4a0 bytes) 000473[0] 1789175918.757258 mapread 0x210825 thru 0x211945 (0x1121 bytes) 000474[1] 1789175918.786264 read 0x775f6d thru 0x777859 (0x18ed bytes) 000477[0] 1789175918.815480 read 0x61e3c6 thru 0x62008d (0x1cc8 bytes) 000478[0] 1789175918.826770 mapwrite 0x4a92ce thru 0x4aa392 (0x10c5 bytes) 000479[1] 1789175918.926951 mapread 0x796ec6 thru 0x7974cf (0x60a bytes) 000481[1] 1789175918.973934 read 0x12c1e6 thru 0x12c4cd (0x2e8 bytes) 000483[1] 1789175918.980019 mapread 0x266000 thru 0x267dc2 (0x1dc3 bytes) 000485[1] 1789175918.992253 write 0x45447d thru 0x455b99 (0x171d bytes) 000487[1] 1789175919.038522 read 0x482519 thru 0x484036 (0x1b1e bytes) 000488[0] 1789175919.048455 read 0x37dedc thru 0x37e927 (0xa4c bytes) 000489[0] 1789175919.076336 punch from 0x5a4836 to 0x5a5c1c (0x13e7 bytes) 000491[0] 1789175919.125077 read 0x6d749b thru 0x6d87c6 (0x132c bytes) 000492[1] 1789175919.141666 write 0x48b424 thru 0x48bba0 (0x77d bytes) 000494[1] 1789175919.185711 trunc from 0x804a59 to 0x09451e (0x77053a bytes) 000495[1] 1789175919.272497 write 0x687280 thru 0x6884e0 (0x1261 bytes) 000497[1] 1789175919.374697 write 0x6cec2d thru 0x6cfbe4 (0xfb8 bytes) 000498[1] 1789175919.379015 read 0x4ee042 thru 0x4ef150 (0x110f bytes) 000500[1] 1789175919.409769 mapread 0x3fa7af thru 0x3fc5ca (0x1e1c bytes) All operations completed A-OK! PASS 16b (39s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:18:47 (1789175927) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:18:52 (1789175932) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.461146 s, 18.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.389288 s, 21.5 MB/s ldlm.namespaces.MGC192.168.204.129@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ddc102cd000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ddc1063b800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ddc1063b800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ddc1063b800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.492 s, 8.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.7528 s, 27.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.23211 s, 47.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.30945 s, 14.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4894 s, 10.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89052 s, 21.4 MB/s PASS 16d (56s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:19:48 (1789175988) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342894 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147111 s, 7.1 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:19:56 (1789175996) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:20:25 (1789176025) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 21:20:54 (1789176054) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 21:21:02 (1789176062) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 21:21:09 (1789176069) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.2353 s, 21.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52913 s, 17.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.08967 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.68771 s, 15.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46156 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.76286 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.95926 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.80408 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87192 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.38742 s, 6.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.51026 s, 5.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56857 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.65007 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.13212 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17984 s, 8.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.62249 s, 5.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.96623 s, 5.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.02541 s, 5.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.1023 s, 5.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:21:56 (1789176116) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 11 21:22:50 EDT 2026 + fsx 2 done: rc=0 Fri Sep 11 21:22:50 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Fri Sep 11 21:22:50 EDT 2026 + fsx 4 done: rc=0 Fri Sep 11 21:22:50 EDT 2026 + fsx 5 done: rc=0 Fri Sep 11 21:22:50 EDT 2026 PASS 16k (63s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:22:59 (1789176179) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 21:23:11 (1789176191) mmap test1: basic mmap operation (PASS, 0.12528s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10546s) mmap test3: concurrent mmap ops on two nodes (PASS, 17.24s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6439s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.32429s) mmap test6: check mmap write/read content on two nodes (PASS, 0.35895s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.6384s) mmap test11: multi-thread shared mmap (PASS, 1.4387s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1519s) PASS 18 (36s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:23:47 (1789176227) oleg429-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:23:52 (1789176232) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:23:59 (1789176239) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 21:24:06 (1789176246) now is 1789176248 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 new atime is 1789176310 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:25:16 (1789176316) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:25:24 (1789176324) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:25:33 (1789176333) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (10s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:25:43 (1789176343) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 21:25:47 (1789176347) utime: good mknod times 1789176348 <= 1789176349 <= 1789176349 for /mnt/lustre/f26a utime: good mknod times 1789176348 <= 1789176349 <= 1789176349 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 21:25:56 (1789176356) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:26:06 (1789176366) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 21:26:08.000000000 -0400 Birth: 2026-09-11 21:26:09.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 21:26:08.000000000 -0400 Birth: 2026-09-11 21:26:09.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:26:18 (1789176378) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:26:30 (1789176390) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:26:38 (1789176398) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.129@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 21:27:02 (1789176422) -rw-r--r-- 1 root root 0 Sep 11 21:27 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:27:12 (1789176432) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43200 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:27:21 (1789176441) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 21:27:30 (1789176450) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 21:27:34 (1789176454) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90076 s, 27.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53679 s, 34.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.12808 s, 16.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.44932 s, 15.2 MB/s contention_seconds 4, contention_events 25 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.37423 s, 22.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.50301 s, 20.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51049 s, 34.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52238 s, 34.4 MB/s PASS 32c (34s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:28:10 (1789176490) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:28:13 (1789176493) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:28:16 (1789176496) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:28:20 (1789176500) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 21:28:23 (1789176503) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ddc102cd000.ost_server_uuid in IDLE state after 0 sec PASS 34 (144s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:30:47 (1789176647) fail_loc=0x80000317 Race attempt 0 Wait for 50510 50522 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50510 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.129@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.129@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 575 (time 1789176665.54 total 10.03 last 57.33) - open/close 1125 (time 1789176675.54 total 20.04 last 54.96) - open/close 1690 (time 1789176685.55 total 30.04 last 56.48) - open/close 2248 (time 1789176695.55 total 40.05 last 55.77) - open/close 2886 (time 1789176705.57 total 50.06 last 63.71) - open/close 3549 (time 1789176715.57 total 60.06 last 66.29) total: 4000 open/close in 69.02 seconds: 57.95 ops/second PASS 35 (77s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:32:04 (1789176724) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04873 s, 50.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.861748 s, 60.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.875775 s, 59.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.736347 s, 71.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.927262 s, 56.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7490560) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.962369 s, 54.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.673994 s, 77.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7485440) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.67827 s, 77.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7490560) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.909127 s, 57.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7488512) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.729202 s, 71.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.641183 s, 81.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7439360) after(7490560) PASS 36 (306s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 21:37:10 (1789177030) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 - create 1261 (time 1789177044.76 total 10.00 last 126.04) - create 2579 (time 1789177054.76 total 20.00 last 131.79) - create 3886 (time 1789177064.76 total 30.01 last 130.67) - create 5044 (time 1789177074.76 total 40.01 last 115.75) - create 6090 (time 1789177084.76 total 50.01 last 104.59) - create 7310 (time 1789177094.77 total 60.01 last 122.00) - create 8481 (time 1789177104.77 total 70.02 last 117.07) - create 9613 (time 1789177114.77 total 80.02 last 113.17) total: 10000 create in 83.15 seconds: 120.27 ops/second PASS 37 (95s) == sanityn test 39a: file mtime does not change after rename ========================================================== 21:38:45 (1789177125) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:38:54 (1789177134) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 21:39:04 (1789177144) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 21:39:12 (1789177152) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 21:39:21 (1789177161) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 21:39:39 (1789177179) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 21:39:56 (1789177196) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 21:40:14 (1789177214) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 21:40:32 (1789177232) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 21:40:47 (1789177247) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 21:41:03 (1789177263) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 21:41:17 (1789177277) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 21:41:30 (1789177290) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 21:41:45 (1789177305) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 21:42:01 (1789177321) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 21:42:14 (1789177334) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 21:42:29 (1789177349) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1085s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:00:34 (1789178434) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:00:47 (1789178447) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:01:02 (1789178462) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:01:17 (1789178477) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:01:31 (1789178491) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:01:46 (1789178506) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:02:01 (1789178521) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:02:14 (1789178534) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:02:28 (1789178548) PASS 43a (79s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:03:47 (1789178627) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:04:02 (1789178642) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:04:16 (1789178656) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:04:30 (1789178670) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:04:44 (1789178684) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:04:59 (1789178699) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:05:13 (1789178713) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:05:29 (1789178729) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:05:33 (1789178733) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (140s) == sanityn test 43k: unlink vs create ==================== 22:07:54 (1789178874) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1294s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:29:28 (1789180168) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 11 22:29:36 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:29:42 (1789180182) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:29:57 (1789180197) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:30:12 (1789180212) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:30:26 (1789180226) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:30:39 (1789180239) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:30:53 (1789180253) Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:31:09 (1789180269) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:31:23 (1789180283) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:31:26 (1789180286) PASS 45a (123s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:33:29 (1789180409) Conflict PASS 45b (16s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:33:45 (1789180425) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:34:01 (1789180441) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:34:15 (1789180455) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:34:30 (1789180470) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:34:44 (1789180484) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:35:00 (1789180500) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:35:15 (1789180515) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 22:35:19 (1789180519) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1371s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 22:58:10 (1789181890) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 22:58:25 (1789181905) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 22:58:40 (1789181920) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (16s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 22:58:56 (1789181936) Conflict PASS 46d (16s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 22:59:12 (1789181952) Conflict PASS 46e (16s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 22:59:28 (1789181968) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 22:59:43 (1789181983) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 22:59:57 (1789181997) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:00:12 (1789182012) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:00:16 (1789182016) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:00:19 (1789182019) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:00:23 (1789182023) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:00:27 (1789182027) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:00:30 (1789182030) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:00:33 (1789182033) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:00:37 (1789182037) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:00:41 (1789182041) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0651443 s, 157 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:00:54 (1789182054) 0+1 records in 0+1 records out 158 bytes copied, 0.0356921 s, 4.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218550) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:01:04 (1789182064) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0388477 s, 26.4 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864326 s, 12.1 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:01:25 (1789182085) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0157768 s, 64.9 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:01:37 (1789182097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480213 s, 21.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:01:46 (1789182106) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 23:01:56 (1789182116) llite.lustre-ffff8ddc102cd000.xattr_cache=1 llite.lustre-ffff8ddc1063b800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:01:57.000000000 -0400 Modify: 2026-09-11 23:01:57.000000000 -0400 Change: 2026-09-11 23:01:57.000000000 -0400 Birth: 2026-09-11 23:01:58.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:02:04.000000000 -0400 Modify: 2026-09-11 23:02:05.000000000 -0400 Change: 2026-09-11 23:02:05.000000000 -0400 Birth: 2026-09-11 23:02:05.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:02:11.000000000 -0400 Modify: 2026-09-11 23:02:11.000000000 -0400 Change: 2026-09-11 23:02:11.000000000 -0400 Birth: 2026-09-11 23:02:11.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:02:11.000000000 -0400 Modify: 2026-09-11 23:02:12.000000000 -0400 Change: 2026-09-11 23:02:12.000000000 -0400 Birth: 2026-09-11 23:02:11.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:02:17.000000000 -0400 Modify: 2026-09-11 23:02:18.000000000 -0400 Change: 2026-09-11 23:02:18.000000000 -0400 Birth: 2026-09-11 23:02:18.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:02:17.000000000 -0400 Modify: 2026-09-11 23:02:18.000000000 -0400 Change: 2026-09-11 23:02:18.000000000 -0400 Birth: 2026-09-11 23:02:18.000000000 -0400 PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 23:02:28 (1789182148) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 23:02:41 (1789182161) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 23:02:55 (1789182175) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 23:03:15 (1789182195) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:03:32 (1789182212) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:03:35 (1789182215) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 23:03:55 (1789182235) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:04:19 (1789182259) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789182261 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.50032 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (131s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:06:30 (1789182390) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789182397 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.00888 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (137s) == sanityn test 60: Verify data_version behaviour ======== 23:08:47 (1789182527) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 23:08:59 (1789182539) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:09:08 (1789182548) PASS 70b (11s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:09:19 (1789182559) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00931622 s, 7.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:09:23 (1789182563) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0140191 s, 4.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:09:27 (1789182567) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:09:30 (1789182570) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0287299 s, 2.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:09:35 (1789182575) llite.lustre-ffff8ddc102cd000.xattr_cache=1 llite.lustre-ffff8ddc1063b800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:09:44 (1789182584) llite.lustre-ffff8ddc102cd000.xattr_cache=1 llite.lustre-ffff8ddc1063b800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:09:53 (1789182593) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0172318 s, 59.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0164141 s, 62.4 kB/s 231386: get lock2 231387: get lock1 231387: done 231387: get lock2 231386: done 231386: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231386: cannot get lock1: Resource deadlock avoided 231387: done 231387: exit rc=0 231386: done 231386: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:10:06 (1789182606) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0613002 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270129 s, 38.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231968 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 23:10:19 (1789182619) opening files: ........ closing files: ........ PASS 76 (78s) == sanityn test 77a: check FIFO NRS policy =============== 23:11:38 (1789182698) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.384539 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589542 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936477 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113243 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12774 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184077 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183969 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165885 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218685 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226158 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114374 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275503 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114309 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192269 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31685 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236828 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306778 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119994 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842632 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116929 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880889 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996317 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125304 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232209 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125906 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803267 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073862 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127367 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500338 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105545 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160296 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138362 s, 75.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370102 s, 28.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 23:11:49 (1789182709) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228838 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438805 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591718 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071986 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119475 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679221 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894791 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150571 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964417 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953731 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130683 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183546 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158489 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226274 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190878 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231368 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254205 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255132 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321044 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555372 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534104 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568492 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607299 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663009 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700568 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687532 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758526 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083332 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092318 s, 11.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706584 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103181 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769829 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169644 s, 6.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.287265 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409839 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100872 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845899 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609837 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379428 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850108 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118793 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185712 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934505 s, 11.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27316 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287605 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130338 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240453 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318427 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267197 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356154 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441095 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790367 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756033 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179986 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238126 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386756 s, 271 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870817 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007241 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079015 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740688 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445282 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381932 s, 27.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381591 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246408 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612969 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269376 s, 38.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 23:12:03 (1789182723) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31955 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719501 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585524 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817202 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758952 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176329 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147341 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209633 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140329 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143413 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101998 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147033 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221022 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206156 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146151 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214095 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248494 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397415 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242016 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288417 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110912 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848469 s, 12.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139029 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565589 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898476 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761189 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183531 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603929 s, 17.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621565 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578681 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463424 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625171 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650277 s, 16.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.268201 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054237 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309741 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488409 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134852 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042267 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177852 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1318 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882126 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712684 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120343 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498093 s, 21.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548731 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143345 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158986 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204709 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250786 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126836 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126026 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133098 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140583 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125499 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108718 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689374 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122864 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1241 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143035 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165208 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862112 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159553 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131749 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144131 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168074 s, 6.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.357146 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574495 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100978 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565454 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846258 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176452 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973138 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916326 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179788 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234436 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270015 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281399 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221079 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174594 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165628 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334247 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252575 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269539 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330532 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138626 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597231 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390726 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477928 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766097 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425875 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491904 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498334 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00890779 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325374 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875137 s, 12.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142218 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797417 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104474 s, 10.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 23:12:20 (1789182740) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.35424 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705903 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112862 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172575 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159727 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089845 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218479 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214793 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317655 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195332 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158588 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151689 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193076 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174443 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212321 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274004 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364284 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651486 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109882 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860172 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133486 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112692 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867295 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117579 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191532 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125788 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120671 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114432 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167658 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109575 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157561 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115378 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162802 s, 6.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.485476 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041155 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505526 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1135 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144395 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670512 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439895 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300564 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934943 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892445 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2493 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389146 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226733 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428649 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300343 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269127 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327212 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699974 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742044 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760644 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174282 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059018 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175966 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182896 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119655 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126186 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150941 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609705 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298547 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485888 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127392 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191126 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588028 s, 17.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.697747 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762866 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799992 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173548 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899306 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176732 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191234 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179294 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185654 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194725 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195304 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203114 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235969 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251603 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217293 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254639 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256395 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807247 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116449 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523993 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561245 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115896 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461691 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546357 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553682 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630521 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636422 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704522 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143288 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748428 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169598 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884126 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12687 s, 8.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (22s) == sanityn test 77e: check TBF NID nrs policy ============ 23:12:41 (1789182761) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.29@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354754 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486553 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435791 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496571 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494555 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161461 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937668 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184724 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188774 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131855 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255257 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342333 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354786 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320497 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333698 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412356 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111247 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295989 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586834 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113743 s, 92.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244204 s, 429 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661701 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106689 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169941 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114791 s, 91.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11742 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173505 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124222 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976994 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136978 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134096 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135963 s, 7.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.540587 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534638 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113157 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456461 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121995 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055531 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175074 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425008 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370643 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139906 s, 7.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294183 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209128 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292391 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25154 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35707 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687227 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222621 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926872 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099749 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702356 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145822 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14584 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108254 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123266 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846089 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130199 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673635 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710348 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147038 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112288 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11533 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121289 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1237 s, 8.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.367882 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576186 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731263 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121916 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154335 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113193 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225655 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21353 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158735 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152593 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186146 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217514 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181385 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21233 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187632 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238806 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332988 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207424 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564679 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849775 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269991 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713118 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453413 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621419 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886992 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519042 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545887 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659783 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746248 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152243 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864084 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803538 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972621 s, 10.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.245992 s, 68.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037116 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324052 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428849 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104226 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147137 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119796 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1561 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200816 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154602 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109846 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180608 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162228 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204071 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249492 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256692 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279283 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546035 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392098 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024308 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554779 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131074 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166175 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167243 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153883 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192409 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110125 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785385 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786761 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178318 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811371 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419309 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344084 s, 30.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (32s) == sanityn test 77f: check TBF JobID nrs policy ========== 23:13:13 (1789182793) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271825 s, 61.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409677 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037341 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752611 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589195 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638829 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741644 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969841 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142218 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694186 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986985 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191983 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136078 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210373 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194245 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237039 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215114 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009039 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520057 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196553 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019924 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791517 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849019 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286601 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046767 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439129 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461338 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573033 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561628 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059339 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796352 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742225 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867547 s, 12.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326568 s, 51.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447667 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481966 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023083 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354093 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143401 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120849 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925324 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990392 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152409 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109647 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100764 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671747 s, 15.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107931 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296652 s, 35.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252426 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215532 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674829 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106898 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292767 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499244 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073516 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018173 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990157 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607757 s, 173 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425567 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450297 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494683 s, 212 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852645 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314114 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356596 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207248 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317001 s, 33.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.269571 s, 62.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036158 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586913 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617451 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102692 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914111 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139439 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172349 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808323 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394939 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17141 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328833 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774167 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915322 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201577 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234262 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25086 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111947 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641396 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103838 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702968 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178155 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883776 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221997 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227328 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495309 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504696 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880492 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455151 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113799 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531489 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612077 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577061 s, 18.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326559 s, 51.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457746 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517472 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166265 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156245 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793362 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934549 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201421 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11682 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15749 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118649 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106288 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841827 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310301 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543456 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275574 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126766 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763606 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571364 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272874 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067818 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408646 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383009 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420883 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761894 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667803 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580179 s, 18.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580627 s, 18.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570002 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877917 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882316 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581396 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392962 s, 267 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 23:13:37 (1789182817) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 23:13:50 (1789182830) oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 PASS 77h (16s) == sanityn test 77i: Change rank of TBF rule ============= 23:14:06 (1789182846) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (28s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 23:14:35 (1789182875) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.672183 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314016 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042353 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040738 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556791 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112116 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167741 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113009 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185194 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185247 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220896 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708787 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193982 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156551 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987122 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170451 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342119 s, 30.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109708 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857176 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623492 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676285 s, 15.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890776 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102081 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106941 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120285 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130567 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179788 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116681 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140182 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152779 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126473 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128671 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147999 s, 7.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.76385 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6016 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (70s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 23:15:44 (1789182944) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2842 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911538 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965651 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14476 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171301 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17188 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122181 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192465 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227615 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255971 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303927 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201716 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190532 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345621 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271313 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207952 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388349 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837642 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062569 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887297 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749657 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671117 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113694 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111717 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086316 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130972 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704205 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904035 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104966 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124823 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108263 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107031 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134685 s, 7.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9872 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9755 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3156 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400596 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728818 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829622 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950756 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255902 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162434 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10369 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182164 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146531 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182217 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207326 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155151 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236188 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844282 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147498 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31514 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035457 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793643 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815514 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070016 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342616 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570401 s, 18.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978599 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799596 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753914 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659176 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679128 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815331 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918267 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123072 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804178 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106037 s, 9.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9944 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9927 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (158s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 23:18:22 (1789183102) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26933 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439738 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146111 s, 71.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104776 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126311 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127824 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105136 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442232 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812839 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796493 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518964 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134843 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18724 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139111 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147261 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203469 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111444 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907066 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109237 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125862 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126368 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880611 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123728 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819844 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129649 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121813 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109295 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125842 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990655 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109529 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105069 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19717 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144731 s, 7.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0398 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8597 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26349 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873103 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843801 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135212 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077922 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119073 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139704 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125268 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924278 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17076 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26479 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154421 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148702 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240758 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240251 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259496 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218516 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076492 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783161 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865545 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067796 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859567 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890564 s, 11.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797589 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987989 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937325 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806616 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706191 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907534 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10392 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111113 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127908 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129161 s, 8.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.099 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9654 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (157s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 23:20:59 (1789183259) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.534753 s, 31.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621378 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828743 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824719 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414081 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156251 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529079 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464703 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19995 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091997 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139261 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261636 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19669 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17338 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683887 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288545 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664684 s, 15.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746854 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425645 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162807 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537679 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352227 s, 29.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463703 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516886 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789506 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782139 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792219 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162206 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124615 s, 84.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115361 s, 90.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823797 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884931 s, 11.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339659 s, 309 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.23278 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90054 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.29@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.29@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.458241 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807166 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854358 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116369 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111775 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119705 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100756 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145401 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17214 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177332 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166424 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192781 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16198 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196484 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159266 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22645 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225726 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433517 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414427 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430863 s, 24.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721872 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042807 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337612 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366484 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381624 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552915 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598386 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579933 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641799 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060153 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915318 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670163 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590007 s, 17.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.36726 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92792 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.29@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815366 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107634 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122444 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189701 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193426 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207278 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277794 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289186 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274847 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169257 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199236 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186296 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35302 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184066 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177721 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272725 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305789 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc102cd000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc102cd000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638849 s, 16.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657728 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137886 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454862 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412822 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550955 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550721 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065145 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516503 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498574 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693433 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062562 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800809 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635013 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916705 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978702 s, 10.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9107 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34123 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662561 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513667 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837856 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16912 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167653 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16984 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110698 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855458 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153734 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178346 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171388 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230996 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27837 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39973 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31016 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32781 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39862 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256521 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584104 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334289 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713001 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322051 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330591 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468922 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586079 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622025 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440649 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559928 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557631 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216494 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711413 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126024 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173668 s, 60.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4002 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99619 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.13168 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91181 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.988839 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756748 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687055 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170153 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199038 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161995 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851882 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235281 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238751 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132981 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352681 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206453 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295494 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168853 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227815 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2405 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295162 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265034 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329263 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331743 s, 31.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372515 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115291 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480598 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121993 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523036 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940562 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641529 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669036 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797263 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983172 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089296 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111176 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946981 s, 11.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8185 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6578 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409106 s, 41.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397185 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274221 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756369 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648959 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164896 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107384 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1316 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172207 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202006 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228828 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149982 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255175 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238125 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202791 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234911 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232266 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731696 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292537 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893057 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881154 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299064 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330216 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577892 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437061 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460256 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052783 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534505 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801406 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581341 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062218 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655719 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681593 s, 15.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.14961 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93005 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270178 s, 62.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364327 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496445 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315575 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270714 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147965 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976888 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117969 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696716 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989978 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140547 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228626 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250377 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189043 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253902 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129349 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184813 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891804 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691482 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025479 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640262 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677668 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548661 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458992 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975526 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537418 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457518 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460479 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588768 s, 17.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127491 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122062 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688343 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707421 s, 14.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7259 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0095 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2996 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94723 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (441s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 23:28:20 (1789183700) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.50473 s, 33.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597429 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786274 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838978 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109897 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148258 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194041 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196577 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193244 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205731 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208232 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197929 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284159 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181769 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248403 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30516 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248009 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150847 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014909 s, 70.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459073 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233221 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226314 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279167 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059719 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644673 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689088 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100146 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427434 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494609 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527937 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551006 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558995 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619854 s, 16.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.96551 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95193 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (49s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 23:29:09 (1789183749) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.29@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.29@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57888 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455117 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070324 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020225 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134777 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917786 s, 11.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101957 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996951 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159774 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155488 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116599 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138138 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120979 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12056 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164561 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226027 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208841 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417911 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856597 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362599 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061364 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429502 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678058 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841529 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046221 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561728 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542339 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083578 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821991 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709656 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988099 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740104 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852109 s, 12.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.16388 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89165 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (52s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 23:30:01 (1789183801) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.29@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.586849 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077639 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078497 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109827 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979442 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166607 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196349 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209956 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218073 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231566 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23226 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231198 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233297 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171765 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150113 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165276 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240687 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412251 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071442 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714091 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327395 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602414 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478688 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767295 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141244 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605027 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976882 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630047 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284244 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392783 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00905489 s, 116 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929215 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516048 s, 20.3 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.20898 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35953 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 23:30:48 (1789183848) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.604176 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458524 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604146 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360837 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221186 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945563 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170645 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727859 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171503 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149567 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709868 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240706 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195461 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189773 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175908 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271605 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709454 s, 14.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685489 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879158 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14079 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521992 s, 20.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101388 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11164 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109174 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129651 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124042 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140374 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120141 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145008 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139449 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160757 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179439 s, 5.8 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0975 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98474 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.8602 s, 11.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95767 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (100s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:32:28 (1789183948) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09074 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369113 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637586 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556992 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177414 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148903 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144379 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125476 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192109 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290046 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23008 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340234 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262662 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286705 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287099 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30051 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40174 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214906 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108135 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679262 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694923 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636356 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409595 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292789 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00684275 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844647 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105969 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495662 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501614 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492104 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291285 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175766 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345706 s, 30.3 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8968 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9607 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (79s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:33:47 (1789184027) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.570093 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694995 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628963 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105074 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121865 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161414 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151034 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182995 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167255 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170936 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188589 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227809 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202163 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233682 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221251 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275209 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348711 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699655 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100098 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630437 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089254 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065619 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102378 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129005 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105297 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12741 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114253 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166056 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980821 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104844 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106364 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926705 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123451 s, 8.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1731 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99347 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg429-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585694 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588462 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436258 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555041 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530681 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574464 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785614 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915214 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145185 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138801 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135338 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17083 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153264 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151811 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133166 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277632 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187616 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295821 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501174 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554071 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571229 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927214 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853947 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073353 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785059 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840657 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865493 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086253 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863216 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178302 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106385 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108643 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118385 s, 8.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1098 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98172 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2947 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92279 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (150s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 23:36:18 (1789184178) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg429-client.virtnet /mnt/lustre (opts:) Stopping client oleg429-client.virtnet /mnt/lustre opts: Stopping clients: oleg429-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg429-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg429-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg429-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.29,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg429-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.29,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg429-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.29,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70125/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.29,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg429-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: kmemleak disabled oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Starting client oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Started clients oleg429-client.virtnet: 192.168.204.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Starting client oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 Started clients oleg429-client.virtnet: 192.168.204.129@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ddc08ea7000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8ddc19da4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc08ea7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc19da4000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.560995 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052405 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571096 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03857 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697215 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139644 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137349 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160748 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23339 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187583 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266211 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235183 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28604 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264571 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327921 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379275 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37632 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285419 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534148 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995382 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756703 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121666 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113086 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724579 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15303 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147742 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964594 s, 10.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708317 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583717 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854638 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746459 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116644 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976662 s, 10.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.88395 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29172 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59067 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647287 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994606 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873632 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135781 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192871 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222168 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220559 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331409 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255457 s, 4.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303686 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262467 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255001 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310521 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303228 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294533 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367813 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816441 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262826 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342382 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373374 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380284 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674807 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174217 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117857 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910748 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130615 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593613 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557737 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322907 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191316 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215308 s, 48.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5053 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98729 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (233s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 23:40:11 (1789184411) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (21s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 23:40:33 (1789184433) On MGS 192.168.204.129, active = nodemap.active=1 On MGS 192.168.204.129, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.54041 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991261 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120891 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133421 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112602 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164074 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248063 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251679 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340962 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340539 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353501 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252858 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247484 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396685 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302766 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358632 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370576 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc08ea7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114953 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814054 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848659 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848802 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967116 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458678 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890214 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499641 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109948 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068459 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123668 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728655 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744593 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769395 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13992 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986617 s, 10.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.48647 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.53639 s, 18.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.129, TBF.id = On MGS 192.168.204.129, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.947178 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103323 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123107 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114706 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653388 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228668 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284469 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279878 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294929 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190106 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267954 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250414 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227355 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204999 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc08ea7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259878 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407117 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437045 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172112 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589363 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455994 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102007 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104402 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100939 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598054 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174846 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284257 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135797 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590781 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736795 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255342 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498298 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192638 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133662 s, 7.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.92275 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.05968 s, 25.8 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg429-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 On MGS 192.168.204.129, active = nodemap.active=1 On MGS 192.168.204.129, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.85245 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090837 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898478 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907255 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134893 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086512 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199187 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2792 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221112 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19849 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229473 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229755 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247186 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295986 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27783 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292934 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333597 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133811 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159476 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159739 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139914 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179215 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159334 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150765 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20457 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183845 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191966 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243773 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206604 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212142 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217175 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256519 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22993 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8ddc08ea7000.lock_unused_count=2 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.247 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9947 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.129, TBF.id = On MGS 192.168.204.129, active = nodemap.active=0 PASS 77kj (192s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 23:43:44 (1789184624) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (14s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 23:43:59 (1789184639) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg429-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.32789 s, 315 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg429-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0642 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 23:44:58 (1789184698) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498479 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566936 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116802 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14791 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513392 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129086 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403159 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841958 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133275 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108004 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289359 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416672 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428307 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407713 s, 2.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516442 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424251 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.566327 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8ddc08ea7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809101 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122821 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615992 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794874 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128923 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872598 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124102 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197546 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21681 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114309 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08249 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214515 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129863 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258674 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc08ea7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193492 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175186 s, 6.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg429-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg429-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.6278 s, 8.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.8627 s, 17.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.886245 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14516 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163293 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146818 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21078 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365941 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340568 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346544 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309204 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324365 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378443 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401961 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435243 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315233 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481594 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466968 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505071 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371826 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41656 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470565 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396515 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369784 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421459 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391645 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406765 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428419 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410911 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413326 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41507 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431151 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436532 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448354 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452493 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8ddc08ea7000.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg429-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg429-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.23079 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47191 s, 19.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (110s) == sanityn test 77o: Changing rank should not panic ====== 23:46:48 (1789184808) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (17s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 23:47:05 (1789184825) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (126s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 23:49:11 (1789184951) mds.MDS.mdt.nrs_policies=tbf oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg429-client: oleg429-server: ssh exited with exit code 36 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg429-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (43s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 23:49:55 (1789184995) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273095 kill md_thread xargs PASS 77r (63s) == sanityn test 77s: Check TBF LRU shrinker ============== 23:50:58 (1789185058) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 7 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (69s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:52:07 (1789185127) oleg429-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg429-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg429-client: oleg429-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 23:52:18 (1789185138) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935707 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:52:20.000000000 -0400 Modify: 2026-09-11 23:52:20.000000000 -0400 Change: 2026-09-11 23:52:20.000000000 -0400 Birth: 2026-09-11 23:52:21.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:52:37 (1789185157) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:52:41 (1789185161) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:52:45 (1789185165) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:52:49 (1789185169) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:52:52 (1789185172) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:52:57 (1789185177) - open/close 596 (time 1789185193.93 total 10.01 last 59.55) - open/close 1224 (time 1789185203.94 total 20.01 last 62.78) - open/close 1788 (time 1789185213.94 total 30.02 last 56.35) - open/close 2329 (time 1789185223.95 total 40.03 last 54.07) total: 2500 open/close in 42.44 seconds: 58.91 ops/second snapshot_time 1789185335.855119428 secs.nsecs start_time 1789185181.432269705 secs.nsecs elapsed_time 154.422849723 secs.nsecs open 2505 samples [usecs] 136 23503 4948728 19235785028 close 2505 samples [usecs] 152 11016 1060715 1022410529 mknod 2501 samples [usecs] 589 23329 4240857 14644634519 unlink 2500 samples [usecs] 768 76731 5687048 35180726728 mkdir 2 samples [usecs] 1239 1374 2613 3422997 rmdir 2 samples [usecs] 6474 16977 23451 330131205 rename 2500 samples [usecs] 549 119533 22618207 481367132147 getattr 3765 samples [usecs] 7 22327 320572 1202729258 parallel_rename_file 2500 samples [usecs] 549 119533 22618207 481367132147 crossdir_rename 2500 samples [usecs] 549 119533 22618207 481367132147 readpage 12 samples [usecs] 6230 73671 249295 9149871155 PASS 81d (162s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:55:39 (1789185339) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7642 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7642 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:55:46 (1789185346) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:55:49 (1789185349) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 23:56:03 (1789185363) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789185365 Finishing test test1 at 1789185374 PASS 85 (17s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:56:20 (1789185380) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 23:56:25 (1789185385) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (5s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:56:30 (1789185390) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:56:35 (1789185395) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5840 0x16d0 0x240000400 1 5800 0x16a8 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5801 0x16a9 0x280000400 0 5841 0x16d1 0x240000400 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 23:56:54 (1789185414) osc.lustre-OST0000-osc-ffff8ddc08ea7000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8ddc19da4000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8ddc08ea7000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8ddc19da4000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.301407 s, 27.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.01544 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290046 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0734019 s, 55.8 kB/s osc.lustre-OST0000-osc-ffff8ddc08ea7000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8ddc19da4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc08ea7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ddc19da4000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:57:10 (1789185430) llite.lustre-ffff8ddc08ea7000.fast_read=0 llite.lustre-ffff8ddc19da4000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0785028 s, 26.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8ddc08ea7000.fast_read=1 llite.lustre-ffff8ddc19da4000.fast_read=1 PASS 95a (24s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:57:33 (1789185453) llite.lustre-ffff8ddc08ea7000.fast_read=0 llite.lustre-ffff8ddc19da4000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0239174 s, 514 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8ddc08ea7000.fast_read=1 llite.lustre-ffff8ddc19da4000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:57:47 (1789185467) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:57:51 (1789185471) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00513643 s, 797 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:57:59 (1789185479) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228413 s, 9.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:58:08 (1789185488) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186595 s, 11.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 23:58:17 (1789185497) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0197377 s, 608 kB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 23:58:25 (1789185505) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0260499 s, 157 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 23:58:34 (1789185514) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137275 s, 298 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 23:58:41 (1789185521) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0163685 s, 250 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 23:58:52 (1789185532) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 7c 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf7c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 23:59:01 (1789185541) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789185544 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789185548 Finishing test test23 at 1789185548 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:59:16 (1789185556) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 23:59:20 (1789185560) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 23:59:48 (1789185588) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:59:52 (1789185592) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0171362 s, 59.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938243 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:59:54.000000000 -0400 Modify: 2026-09-11 23:59:54.000000000 -0400 Change: 2026-09-11 23:59:54.000000000 -0400 Birth: 2026-09-11 23:59:54.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938243 -rw-r--r-- 1 0 0 1789185594 1789185594 1789185594 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938243 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:59:54.000000000 -0400 Modify: 2026-09-11 23:59:54.000000000 -0400 Change: 2026-09-11 23:59:54.000000000 -0400 Birth: 2026-09-11 23:59:54.000000000 -0400 1789185594 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 00:00:01 (1789185601) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 00:00:09 (1789185609) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286102 s, 36.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5808 0x16b0 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:00:20 (1789185620) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.37503 s, 28.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5850 0x16da 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7642 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 00:00:35 (1789185635) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00507276 s, 2.0 MB/s llite.lustre-ffff8ddc08ea7000.hybrid_io=0 llite.lustre-ffff8ddc19da4000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.84802 s, 1.7 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:00:48 (1789185648) 192.168.204.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: kmemleak disabled oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre2 PASS 109 (50s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:01:38 (1789185698) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 00:01:40.000000000 -0400 Modify: 2026-09-12 00:01:40.000000000 -0400 Change: 2026-09-12 00:01:40.000000000 -0400 Birth: 2026-09-12 00:01:41.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg429-client.virtnet oleg429-client.virtnet PASS 110 (95s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:03:14 (1789185794) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 00:03:19 (1789185799) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 00:03:22 (1789185802) lustre-OST0000-osc-ffff8ddc03dc5800 active. lustre-OST0001-osc-ffff8ddc03dc5800 active. lustre-OST0000-osc-ffff8ddc03dc5800 active. lustre-OST0001-osc-ffff8ddc03dc5800 active. MGC192.168.204.129@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 00:03:31 (1789185811) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:03:35 (1789185815) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:03:39 (1789185819) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 00:03:43 (1789185823) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0467267 s, 438 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.122988 s, 167 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (12s) == sanityn test 122: directory size is consistent across mounts ========================================================== 00:03:55 (1789185835) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:04:06 (1789185846) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=10 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1582 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1974 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1512 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1776 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1839 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1345 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1458 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2121 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1151 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.152 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1092 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1466 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0985 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1153 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1444 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.0818 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.0991 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3553 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2541 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ddc03dc5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ddc03dc5800.ost_server_uuid in FULL state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 10583 sec ============= 00:05:34 (1789185934) === sanityn: start cleanup 00:05:36 (1789185936) === Stopping clients: oleg429-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg429-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:10:05 (1789186205) ===