-----============= acceptance-small: sanityn ============----- Mon Aug 10 00:54:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg437-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 00:55:02 (1786337702) === Starting client oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Mount client oleg437-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Started clients oleg437-client.virtnet: 192.168.204.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg437-client.virtnet: executing check_config_client /mnt/lustre oleg437-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg437-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f34c3021800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f34c558a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c3021800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c558a800.idle_timeout=debug disable quota as required oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 00:55:45 (1786337745) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570381 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 00:55:48 (1786337748) PASS 0 (12s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 00:56:00 (1786337760) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 00:56:10 (1786337770) -rw-r--r-- 1 root root 0 Aug 10 00:56 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (11s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 00:56:20 (1786337780) -rw-r--r-- 1 root root 0 Aug 10 00:56 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (9s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 00:56:30 (1786337790) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (9s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 00:56:38 (1786337798) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 00:56:47 (1786337807) total 1 -rwxrwxrwx 1 root root 0 Aug 10 00:56 f2a -rwxrwxrwx 1 root root 0 Aug 10 00:56 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 10 00:56 f2a -rwxrwxrwx 1 root root 0 Aug 10 00:56 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0122313 s, 41.9 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 00:56:56 (1786337816) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 00:57:01 (1786337821) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.708443 s, 3.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (11s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 00:57:12 (1786337832) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 00:57:19 (1786337839) PASS 4 (11s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 00:57:31 (1786337851) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (12s) == sanityn test 6: remove of open file on other node ============================================================================ 00:57:43 (1786337863) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 00:57:52 (1786337872) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (11s) == sanityn test 8: remove of open special file on other node ==================================================================== 00:58:03 (1786337883) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (10s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 00:58:13 (1786337893) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 00:58:22 (1786337902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110544 s, 9.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0726971 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 00:58:31 (1786337911) 1+0 records in 1+0 records out 1 byte copied, 0.0117071 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0192636 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102337 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132051 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170663 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017284 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010555 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111823 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0220011 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011312 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0224734 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0218967 s, 0.0 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 00:58:41 (1786337921) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0249744 s, 123 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0843356 s, 48.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000596319 s, 5.2 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 00:58:50 (1786337930) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (10s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 00:59:00 (1786337940) warning: '-runtime' deprecated, use '-t runtime' instead - link 277 (time 1786337953.60 total 10.07 last 27.52) - link 523 (time 1786337963.63 total 20.09 last 24.55) - link 790 (time 1786337973.63 total 30.09 last 26.69) - link 1073 (time 1786337983.66 total 40.12 last 28.22) - link 1325 (time 1786337993.68 total 50.14 last 25.15) - link 1601 (time 1786338003.70 total 60.16 last 27.55) - link 1857 (time 1786338013.70 total 70.16 last 25.60) - link 2071 (time 1786338023.70 total 80.16 last 21.40) - link 2293 (time 1786338033.72 total 90.18 last 22.15) total: 2525 link in 100.02 seconds: 25.25 ops/second using seed 2029852811 running for 100 seconds - stat 3118 (time 1786337955 ; total 11 ; last 11) - stat 6741 (time 1786337966 ; total 22 ; last 11) - stat 10000 (time 1786337976 ; total 32 ; last 10) - stat 13849 (time 1786337987 ; total 43 ; last 11) - stat 16753 (time 1786337998 ; total 54 ; last 11) - stat 18503 (time 1786338009 ; total 65 ; last 11) - stat 19991 (time 1786338020 ; total 76 ; last 11) - stat 20000 (time 1786338020 ; total 76 ; last 0) - stat 21682 (time 1786338031 ; total 87 ; last 11) - stat 23048 (time 1786338042 ; total 98 ; last 11) total: 23362 stats in 100 seconds: 233.619995 stats/second - unlinked 0 (time 1786337994 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2525) error: No such file or directory total: 2524 unlinks in 87 seconds: 29.011494 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21434) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21437) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 01:01:28 (1786338088) f13 f13 PASS 13 (13s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:01:41 (1786338101) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 22638 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (11s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:01:52 (1786338112) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 23220 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (10s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:02:02 (1786338122) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23800 Terminated $DIR1/$tdir/sleep 60 PASS 14b (10s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:02:12 (1786338132) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24384 Terminated $DIR1/$tdir/sleep 60 PASS 14c (10s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:02:22 (1786338142) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24968 Terminated $DIR1/$tdir/sleep 60 PASS 14d (12s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:02:34 (1786338154) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (59s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:03:33 (1786338213) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.725554 s, 11.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.369402 s, 22.7 MB/s Chance of close/open is 1 in 50 Seed set to 8217 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000003[0] 1786338218.619752 write 0x9208ce thru 0x925d9b (0x54ce bytes) 000007[0] 1786338218.685540 read 0x5617d0 thru 0x56313e (0x196f bytes) 000010[0] 1786338218.705015 mapread 0x22740a thru 0x229735 (0x232c bytes) 000012[1] 1786338218.788454 write 0x9b3fc7 thru 0x9c2486 (0xe4c0 bytes) 000014[1] 1786338218.838763 mapwrite 0x0bf720 thru 0x0cd3db (0xdcbc bytes) truncating to largest ever: 0x24d377 000015[1] 1786338218.930859 trunc from 0x9c2487 to 0x24d376 (0x775110 bytes) 000016[1] 1786338218.955955 write 0x0f8ea5 thru 0x0fc9b7 (0x3b13 bytes) truncating to largest ever: 0x555904 000017[1] 1786338218.996390 trunc from 0x24d377 to 0x555903 (0x30858d bytes) 000018[0] 1786338219.032830 mapwrite 0x3e1f3a thru 0x3e7163 (0x522a bytes) truncating to largest ever: 0x7e3161 000019[0] 1786338219.213865 trunc from 0x555904 to 0x7e3160 (0x28d85d bytes) truncating to largest ever: 0x816ae3 000020[0] 1786338219.261103 trunc from 0x7e3161 to 0x816ae2 (0x33982 bytes) 000028[0] 1786338219.341068 read 0x21cd16 thru 0x21d782 (0xa6d bytes) 000029[1] 1786338219.361417 trunc from 0x816ae3 to 0x08b924 (0x78b1be bytes) 000034[1] 1786338219.438894 read 0x035a2e thru 0x0433ca (0xd99d bytes) 000035[1] 1786338219.450581 read 0x07abf2 thru 0x07fc39 (0x5048 bytes) 000039[0] 1786338219.489280 mapread 0x032875 thru 0x0412c9 (0xea55 bytes) 000040[0] 1786338219.539480 mapwrite 0x8fc988 thru 0x8fdef9 (0x1572 bytes) 000041[0] 1786338219.627822 write 0x76d97c thru 0x7765cf (0x8c54 bytes) 000043[1] 1786338219.650788 mapread 0x6e6619 thru 0x6f24b2 (0xbe9a bytes) 000044[1] 1786338219.755361 trunc from 0x8fdefa to 0x0ab7fa (0x8526ff bytes) 000045[1] 1786338219.781971 mapread 0x04fb04 thru 0x05ca80 (0xcf7d bytes) 000046[0] 1786338219.851931 trunc from 0x0ab7fb to 0x6d4b5b (0x629361 bytes) 000047[1] 1786338219.879052 write 0x558027 thru 0x564815 (0xc7ef bytes) 000049[1] 1786338219.919914 write 0x910154 thru 0x91be59 (0xbd06 bytes) 000051[1] 1786338219.947798 trunc from 0x91be5a to 0x09ce36 (0x87f023 bytes) 000054[0] 1786338219.977611 mapread 0x003f0d thru 0x007f31 (0x4025 bytes) 000055[1] 1786338220.002326 write 0x529664 thru 0x52aca4 (0x1641 bytes) 000056[1] 1786338220.006438 trunc from 0x52aca5 to 0x248d3c (0x2e1f68 bytes) 000057[0] 1786338220.044635 read 0x07bf6d thru 0x07c488 (0x51c bytes) 000059[1] 1786338220.144610 write 0x279049 thru 0x286f53 (0xdf0b bytes) truncating to largest ever: 0x86903f 000065[1] 1786338220.441091 trunc from 0x286f54 to 0x86903e (0x5e20eb bytes) 000068[1] 1786338220.459696 trunc from 0x86903f to 0x4eafeb (0x37e053 bytes) 000069[1] 1786338220.467099 trunc from 0x4eafec to 0x657a7b (0x16ca90 bytes) truncating to largest ever: 0x9db4e7 000070[1] 1786338220.478552 trunc from 0x657a7c to 0x9db4e6 (0x383a6b bytes) 000071[0] 1786338220.520621 trunc from 0x9db4e7 to 0x114c61 (0x8c6885 bytes) 000073[1] 1786338220.591778 write 0x83f5ff thru 0x84b371 (0xbd73 bytes) 000074[1] 1786338220.634342 mapread 0x338480 thru 0x341430 (0x8fb1 bytes) 000075[0] 1786338220.682621 mapwrite 0x1478f3 thru 0x154a3c (0xd14a bytes) 000076[0] 1786338220.859447 read 0x471a01 thru 0x47ec8a (0xd28a bytes) 000077[0] 1786338220.863812 mapwrite 0x29d307 thru 0x2a5617 (0x8311 bytes) 000079[0] 1786338220.926959 mapread 0x628fe9 thru 0x6328e7 (0x98ff bytes) 000080[1] 1786338220.989548 trunc from 0x84b372 to 0x695902 (0x1b5a6f bytes) 000082[0] 1786338221.053530 mapread 0x089915 thru 0x093366 (0x9a52 bytes) 000083[1] 1786338221.085953 mapwrite 0x22a2f0 thru 0x235409 (0xb11a bytes) 000085[1] 1786338221.204777 write 0x060e0b thru 0x061ff1 (0x11e7 bytes) 000088[0] 1786338221.280798 write 0x6be378 thru 0x6c6bba (0x8843 bytes) 000089[0] 1786338221.302677 mapwrite 0x9e9b96 thru 0x9ed0e0 (0x354b bytes) 000090[0] 1786338221.387967 mapwrite 0x69f40e thru 0x6aa4f8 (0xb0eb bytes) 000091[0] 1786338221.447005 read 0x490543 thru 0x4962b9 (0x5d77 bytes) 000092[1] 1786338221.456026 write 0x6751a9 thru 0x684691 (0xf4e9 bytes) 000097[0] 1786338221.560340 mapwrite 0x73049b thru 0x73ec5d (0xe7c3 bytes) 000098[1] 1786338221.639619 trunc from 0x9ed0e1 to 0x27b7b9 (0x771927 bytes) 000099[0] 1786338221.686104 trunc from 0x27b7ba to 0x32003a (0xa4881 bytes) 000101[0] 1786338221.722383 write 0x1d9edc thru 0x1e05a6 (0x66cb bytes) 000104[0] 1786338221.756402 trunc from 0x32003b to 0x4c1a37 (0x1a19fd bytes) 000105[0] 1786338221.770524 mapwrite 0x42a2ad thru 0x4304fa (0x624e bytes) 000106[0] 1786338221.836584 read 0x2cf785 thru 0x2d575f (0x5fdb bytes) 000107[0] 1786338221.862318 trunc from 0x4c1a38 to 0x716639 (0x254c02 bytes) 000109[1] 1786338221.884850 trunc from 0x71663a to 0x33ddf5 (0x3d8844 bytes) 000110[0] 1786338221.963327 write 0x8651cd thru 0x868653 (0x3487 bytes) 000111[0] 1786338221.998782 mapwrite 0x412995 thru 0x416d44 (0x43b0 bytes) 000112[1] 1786338222.066707 trunc from 0x868654 to 0x2878d0 (0x5e0d83 bytes) 000114[1] 1786338222.098867 write 0x8e33dd thru 0x8ed87b (0xa49f bytes) 000115[0] 1786338222.119331 read 0x72c75b thru 0x730bd9 (0x447f bytes) 000119[0] 1786338222.244094 write 0x754aa0 thru 0x763626 (0xeb87 bytes) 000120[1] 1786338222.267443 write 0x08aea4 thru 0x094f0c (0xa069 bytes) 000121[1] 1786338222.289573 mapwrite 0x012481 thru 0x01e848 (0xc3c8 bytes) 000124[1] 1786338222.462161 read 0x5502a6 thru 0x55d87e (0xd5d9 bytes) 000128[1] 1786338222.501250 trunc from 0x8ed87c to 0x3c06be (0x52d1bd bytes) 000130[0] 1786338222.535580 mapread 0x010d47 thru 0x020999 (0xfc53 bytes) 000132[0] 1786338222.580391 read 0x0ffee1 thru 0x10b426 (0xb546 bytes) 000133[1] 1786338222.627927 read 0x3b0eee thru 0x3b6af1 (0x5c04 bytes) 000136[0] 1786338222.710681 mapwrite 0x508855 thru 0x50ed48 (0x64f4 bytes) 000137[1] 1786338222.811728 read 0x35335d thru 0x35d8f0 (0xa594 bytes) 000139[1] 1786338222.873721 trunc from 0x50ed49 to 0x5ef72f (0xe09e7 bytes) 000140[0] 1786338222.915486 trunc from 0x5ef730 to 0x48ae32 (0x1648fd bytes) 000144[1] 1786338222.964267 mapread 0x1aafdf thru 0x1af670 (0x4692 bytes) 000146[1] 1786338223.065731 write 0x0e29fb thru 0x0e58c4 (0x2eca bytes) 000147[1] 1786338223.187537 mapwrite 0x50eba2 thru 0x516cb0 (0x810f bytes) 000148[0] 1786338223.907610 mapread 0x33cc94 thru 0x34caa9 (0xfe16 bytes) 000150[1] 1786338224.321634 mapwrite 0x2e079d thru 0x2ebe91 (0xb6f5 bytes) 000151[1] 1786338224.736018 trunc from 0x516cb1 to 0x3cbc58 (0x14b058 bytes) 000152[1] 1786338224.870323 mapread 0x38d09e thru 0x39cdbf (0xfd22 bytes) 000153[1] 1786338225.049437 mapread 0x1a8fdb thru 0x1b6a70 (0xda96 bytes) 000155[1] 1786338225.210906 mapread 0x243879 thru 0x24742b (0x3bb3 bytes) 000156[1] 1786338225.280743 mapread 0x3750a6 thru 0x37712b (0x2086 bytes) 000158[1] 1786338225.346620 trunc from 0x3cbc59 to 0x4fef2d (0x1332d5 bytes) 000161[1] 1786338225.558652 mapwrite 0x54896a thru 0x54b430 (0x2ac7 bytes) 000162[0] 1786338225.707160 mapread 0x435af6 thru 0x43cf92 (0x749d bytes) 000164[0] 1786338225.798702 mapread 0x1d6f59 thru 0x1de2c8 (0x7370 bytes) 000166[0] 1786338225.891989 mapread 0x163f8d thru 0x169f33 (0x5fa7 bytes) 000168[1] 1786338225.934212 write 0x651c77 thru 0x660ddf (0xf169 bytes) 000170[1] 1786338225.979396 trunc from 0x660de0 to 0x7daa69 (0x179c8a bytes) 000172[0] 1786338226.071481 read 0x664e8b thru 0x66a9cc (0x5b42 bytes) 000176[1] 1786338226.166025 write 0x883cb0 thru 0x8901f4 (0xc545 bytes) 000177[0] 1786338226.188003 mapwrite 0x4f8e1a thru 0x507598 (0xe77f bytes) 000178[1] 1786338226.343994 trunc from 0x8901f5 to 0x7c4ccb (0xcb529 bytes) 000179[1] 1786338226.408054 mapwrite 0x0abaef thru 0x0ace97 (0x13a9 bytes) 000181[0] 1786338226.524383 write 0x6ef0b5 thru 0x6f5e99 (0x6de5 bytes) 000183[1] 1786338226.602112 mapread 0x71599f thru 0x724a82 (0xf0e4 bytes) 000185[1] 1786338226.701582 write 0x47d2af thru 0x481954 (0x46a6 bytes) 000186[0] 1786338226.820989 write 0x9c4423 thru 0x9c48aa (0x488 bytes) 000187[0] 1786338226.942021 mapread 0x6e4333 thru 0x6eb3f9 (0x70c7 bytes) 000189[1] 1786338227.152958 mapwrite 0x9905d8 thru 0x99f03f (0xea68 bytes) 000193[1] 1786338227.508877 mapread 0x26a1fb thru 0x26dcfc (0x3b02 bytes) 000194[1] 1786338227.554496 mapwrite 0x1c1e23 thru 0x1c7d2a (0x5f08 bytes) 000195[1] 1786338227.630481 mapread 0x805064 thru 0x809f5b (0x4ef8 bytes) 000199[0] 1786338227.699901 mapwrite 0x4bfcd6 thru 0x4ce6a7 (0xe9d2 bytes) 000201[0] 1786338227.855111 trunc from 0x9c48ab to 0x280cd3 (0x743bd7 bytes) 000205[0] 1786338227.971410 read 0x23e7f1 thru 0x2476c9 (0x8ed9 bytes) 000206[0] 1786338228.036076 trunc from 0x280cd4 to 0x4c9981 (0x248cae bytes) 000208[0] 1786338228.090057 write 0x3ca64b thru 0x3d84b3 (0xde69 bytes) 000209[0] 1786338228.163890 mapread 0x4b0e8b thru 0x4bc574 (0xb6ea bytes) 000210[1] 1786338228.166543 mapwrite 0x2db91c thru 0x2e8497 (0xcb7c bytes) 000211[1] 1786338228.340509 write 0x03c00c thru 0x043eae (0x7ea3 bytes) 000212[0] 1786338228.370497 read 0x43bc21 thru 0x44b7d1 (0xfbb1 bytes) 000213[0] 1786338228.395456 read 0x141413 thru 0x1450b5 (0x3ca3 bytes) 000215[0] 1786338228.469846 trunc from 0x4c9982 to 0x71a60a (0x250c89 bytes) 000220[0] 1786338228.550055 write 0x8b39a3 thru 0x8bc5bf (0x8c1d bytes) 000221[1] 1786338228.594520 mapread 0x116976 thru 0x117384 (0xa0f bytes) 000224[0] 1786338228.677099 mapread 0x46c3d1 thru 0x479717 (0xd347 bytes) 000226[1] 1786338228.702539 read 0x5be86b thru 0x5ca8ee (0xc084 bytes) 000228[1] 1786338228.746022 write 0x6d257f thru 0x6d4f60 (0x29e2 bytes) 000230[0] 1786338228.806095 mapread 0x15c6f2 thru 0x169d53 (0xd662 bytes) 000232[0] 1786338228.894049 trunc from 0x8bc5c0 to 0x88019f (0x3c420 bytes) 000233[0] 1786338228.977903 mapread 0x76f89c thru 0x776840 (0x6fa5 bytes) 000234[1] 1786338229.035961 mapread 0x768227 thru 0x76afe2 (0x2dbc bytes) 000235[0] 1786338229.105598 write 0x7861af thru 0x78c38a (0x61dc bytes) 000238[1] 1786338229.142894 read 0x686944 thru 0x68c5c3 (0x5c80 bytes) 000240[0] 1786338229.217530 trunc from 0x8801a0 to 0x3549e1 (0x52b7be bytes) 000244[0] 1786338229.398842 trunc from 0x3549e2 to 0x066b80 (0x2ede61 bytes) 000250[1] 1786338229.571861 trunc from 0x066b81 to 0x099159 (0x325d9 bytes) 000251[0] 1786338229.669958 mapwrite 0x047fcb thru 0x04967b (0x16b1 bytes) 000252[1] 1786338229.795508 mapread 0x017509 thru 0x01839b (0xe93 bytes) 000254[0] 1786338229.840836 read 0x013dd4 thru 0x01c255 (0x8482 bytes) 000255[1] 1786338229.878993 read 0x05066b thru 0x05de3f (0xd7d5 bytes) 000256[1] 1786338229.927929 mapwrite 0x2934a4 thru 0x29496c (0x14c9 bytes) 000257[0] 1786338230.101162 read 0x263cb7 thru 0x268815 (0x4b5f bytes) 000258[1] 1786338230.228544 write 0x55f1c2 thru 0x567d79 (0x8bb8 bytes) 000259[0] 1786338230.270098 read 0x2fb66b thru 0x30795f (0xc2f5 bytes) 000260[0] 1786338230.402050 mapwrite 0x64a49d thru 0x651d5a (0x78be bytes) 000261[1] 1786338230.617416 mapwrite 0x34e1f3 thru 0x35658e (0x839c bytes) 000262[0] 1786338230.758983 mapwrite 0x971c85 thru 0x9746b7 (0x2a33 bytes) 000263[0] 1786338230.906876 mapwrite 0x83bbe7 thru 0x83fee6 (0x4300 bytes) 000264[1] 1786338231.014857 trunc from 0x9746b8 to 0x197284 (0x7dd433 bytes) 000266[1] 1786338231.128478 read 0x0692dc thru 0x0729bb (0x96e0 bytes) 000267[1] 1786338231.169993 mapread 0x00dc2d thru 0x01871e (0xaaf2 bytes) 000270[1] 1786338231.251164 mapread 0x150649 thru 0x15be04 (0xb7bc bytes) 000273[1] 1786338231.285724 read 0x13a9ad thru 0x14218b (0x77df bytes) 000274[0] 1786338231.313628 read 0x09bc7e thru 0x09f584 (0x3907 bytes) 000277[0] 1786338231.460892 write 0x06dca5 thru 0x0787aa (0xab06 bytes) 000279[0] 1786338231.530933 read 0x163caa thru 0x16ac0d (0x6f64 bytes) 000280[1] 1786338231.606455 trunc from 0x197285 to 0x5d79cd (0x440749 bytes) 000282[1] 1786338231.774167 read 0x1b527e thru 0x1bc248 (0x6fcb bytes) 000284[1] 1786338231.820015 mapread 0x01ea08 thru 0x02dbda (0xf1d3 bytes) 000285[0] 1786338231.869888 write 0x55485d thru 0x5600ea (0xb88e bytes) 000286[1] 1786338231.912517 mapwrite 0x48d81f thru 0x48ebe4 (0x13c6 bytes) 000291[1] 1786338232.288826 mapread 0x3f36f3 thru 0x3f86de (0x4fec bytes) 000297[0] 1786338232.451997 mapwrite 0x180250 thru 0x18394a (0x36fb bytes) 000299[0] 1786338232.728079 mapread 0x0e29cf thru 0x0e5344 (0x2976 bytes) 000300[0] 1786338232.793772 write 0x4cd955 thru 0x4d7686 (0x9d32 bytes) 000302[1] 1786338232.946760 mapread 0x067b4a thru 0x073248 (0xb6ff bytes) 000304[1] 1786338233.059039 write 0x96323f thru 0x964fde (0x1da0 bytes) 000307[1] 1786338233.148616 write 0x22c398 thru 0x23bae6 (0xf74f bytes) 000308[0] 1786338233.179151 read 0x4eef2d thru 0x4f0d71 (0x1e45 bytes) 000309[1] 1786338233.199543 trunc from 0x964fdf to 0x94b092 (0x19f4c bytes) 000310[0] 1786338233.295836 read 0x33d7d6 thru 0x347812 (0xa03d bytes) 000311[1] 1786338233.358637 read 0x35d71c thru 0x3692b8 (0xbb9d bytes) 000313[1] 1786338233.383558 write 0x30a124 thru 0x313f60 (0x9e3d bytes) 000314[1] 1786338233.427154 write 0x293b25 thru 0x29a844 (0x6d20 bytes) 000317[1] 1786338233.476489 mapwrite 0x3b5b2a thru 0x3bda6f (0x7f46 bytes) 000319[1] 1786338233.606100 trunc from 0x94b093 to 0x19d958 (0x7ad73a bytes) 000320[1] 1786338233.687906 write 0x4af3b2 thru 0x4bc9c7 (0xd616 bytes) 000328[1] 1786338233.860918 mapwrite 0x4b1f7e thru 0x4c1230 (0xf2b3 bytes) 000329[0] 1786338233.984416 trunc from 0x4c1231 to 0x926efd (0x465ccd bytes) 000330[1] 1786338234.096740 trunc from 0x926efe to 0x73eca3 (0x1e825a bytes) 000333[1] 1786338234.182919 mapwrite 0x2cc127 thru 0x2d5700 (0x95da bytes) 000334[1] 1786338234.374847 write 0x37f37b thru 0x38e2d4 (0xef5a bytes) 000335[1] 1786338234.429709 trunc from 0x73eca4 to 0x20395c (0x53b347 bytes) 000343[0] 1786338234.512343 trunc from 0x20395d to 0x93468f (0x730d33 bytes) 000348[1] 1786338234.741842 trunc from 0x934690 to 0x50a03a (0x42a655 bytes) 000349[1] 1786338234.799088 write 0x1c7f2d thru 0x1cdbe7 (0x5cbb bytes) 000351[1] 1786338234.855403 trunc from 0x50a03b to 0x214977 (0x2f56c3 bytes) 000353[1] 1786338234.918732 mapread 0x0e91c2 thru 0x0f274a (0x9589 bytes) 000354[1] 1786338234.986099 read 0x20743f thru 0x20ce6e (0x5a30 bytes) 000355[1] 1786338235.068037 write 0x876326 thru 0x879e64 (0x3b3f bytes) 000356[0] 1786338235.077838 mapread 0x39afae thru 0x39bad7 (0xb2a bytes) 000359[0] 1786338235.163465 mapwrite 0x57d4c7 thru 0x57e35e (0xe98 bytes) 000360[1] 1786338235.238024 write 0x455f9b thru 0x45e2c9 (0x832f bytes) 000362[0] 1786338235.365139 read 0x0eaf78 thru 0x0f6abb (0xbb44 bytes) 000363[1] 1786338235.387325 mapwrite 0x5ee240 thru 0x5f67cc (0x858d bytes) 000367[1] 1786338235.696053 mapwrite 0x77b353 thru 0x77fa4b (0x46f9 bytes) 000369[1] 1786338235.860874 read 0x23957e thru 0x24190d (0x8390 bytes) 000371[1] 1786338235.908626 write 0x2bcace thru 0x2cb08d (0xe5c0 bytes) 000372[0] 1786338235.951123 mapread 0x503e64 thru 0x5095c1 (0x575e bytes) 000374[0] 1786338236.043065 read 0x1df7c3 thru 0x1e1d01 (0x253f bytes) 000376[1] 1786338236.093609 read 0x69e58e thru 0x6ab49f (0xcf12 bytes) 000378[1] 1786338236.120518 write 0x4a5133 thru 0x4a587a (0x748 bytes) 000380[1] 1786338236.170906 mapread 0x4cf765 thru 0x4d0b89 (0x1425 bytes) 000381[0] 1786338236.208365 read 0x1e15ca thru 0x1e2961 (0x1398 bytes) 000383[1] 1786338236.216911 trunc from 0x879e65 to 0x3e9553 (0x490911 bytes) 000386[0] 1786338236.380770 trunc from 0x3e9554 to 0x45b832 (0x722df bytes) 000392[1] 1786338236.578972 read 0x149c41 thru 0x14e1df (0x459f bytes) 000395[0] 1786338236.713111 read 0x1ed689 thru 0x1f605f (0x89d7 bytes) 000396[0] 1786338236.731854 write 0x5b47f7 thru 0x5c0315 (0xbb1f bytes) 000401[1] 1786338236.803089 mapwrite 0x3f4ab4 thru 0x3fb1be (0x670b bytes) 000402[0] 1786338236.920050 read 0x060149 thru 0x061d2f (0x1be7 bytes) 000406[1] 1786338236.971852 read 0x46bd88 thru 0x477d70 (0xbfe9 bytes) 000408[0] 1786338237.020954 trunc from 0x5c0316 to 0x7aa10b (0x1e9df6 bytes) 000413[0] 1786338237.142673 read 0x10935e thru 0x114ab1 (0xb754 bytes) 000414[0] 1786338237.156367 trunc from 0x7aa10c to 0x1d2877 (0x5d7894 bytes) 000416[0] 1786338237.245919 write 0x0e1938 thru 0x0e43a9 (0x2a72 bytes) 000419[1] 1786338237.290651 mapwrite 0x4fff3e thru 0x505052 (0x5115 bytes) 000421[1] 1786338237.444156 mapwrite 0x4fb160 thru 0x506f69 (0xbe0a bytes) 000423[0] 1786338237.614146 write 0x848558 thru 0x84ab92 (0x263b bytes) 000424[0] 1786338237.680003 mapread 0x54cceb thru 0x554b5c (0x7e72 bytes) 000428[1] 1786338237.833086 mapread 0x6b4dfb thru 0x6b8181 (0x3387 bytes) 000430[1] 1786338237.953853 trunc from 0x84ab93 to 0x1d814d (0x672a45 bytes) 000431[1] 1786338238.019733 trunc from 0x1d814e to 0x1cc880 (0xb8cd bytes) 000432[0] 1786338238.078360 read 0x170bb0 thru 0x17a074 (0x94c5 bytes) 000436[1] 1786338238.196787 mapread 0x04eca1 thru 0x0534bc (0x481c bytes) 000437[1] 1786338238.220954 write 0x4ccb1d thru 0x4dc195 (0xf679 bytes) 000438[0] 1786338238.243297 read 0x1f0a2a thru 0x1fd3f0 (0xc9c7 bytes) 000439[0] 1786338238.382116 read 0x46a32b thru 0x46c301 (0x1fd7 bytes) 000441[1] 1786338238.455195 mapread 0x359cfa thru 0x35edee (0x50f5 bytes) 000442[0] 1786338238.508989 mapwrite 0x5e6072 thru 0x5f00b7 (0xa046 bytes) 000444[1] 1786338238.848145 write 0x390ef4 thru 0x39fa5f (0xeb6c bytes) 000446[1] 1786338238.933585 read 0x52341c thru 0x525b26 (0x270b bytes) 000451[0] 1786338238.991943 mapread 0x351571 thru 0x360f59 (0xf9e9 bytes) 000452[0] 1786338239.088978 read 0x5794ee thru 0x580081 (0x6b94 bytes) 000453[0] 1786338239.118286 mapwrite 0x7b8492 thru 0x7bd9bf (0x552e bytes) 000455[0] 1786338239.339929 mapwrite 0x3d5f16 thru 0x3e16e3 (0xb7ce bytes) 000457[0] 1786338239.513947 write 0x6427c0 thru 0x650fca (0xe80b bytes) 000458[0] 1786338239.529139 read 0x4fab48 thru 0x4fb9ac (0xe65 bytes) 000461[1] 1786338239.624389 trunc from 0x7bd9c0 to 0x0d5e55 (0x6e7b6a bytes) 000464[1] 1786338239.709812 write 0x13608a thru 0x145e0a (0xfd81 bytes) 000467[1] 1786338239.747948 mapwrite 0x1ce4f4 thru 0x1d2478 (0x3f85 bytes) 000471[1] 1786338239.994683 mapread 0x14f438 thru 0x15ddda (0xe9a3 bytes) 000472[1] 1786338240.021096 mapread 0x05d214 thru 0x0607c2 (0x35af bytes) 000473[0] 1786338240.095945 mapread 0x1bc43c thru 0x1c9329 (0xceee bytes) 000474[0] 1786338240.186615 mapread 0x0431a0 thru 0x050bc4 (0xda25 bytes) 000476[1] 1786338240.258666 mapread 0x1a00ac thru 0x1a57da (0x572f bytes) 000478[1] 1786338240.311998 mapread 0x12b945 thru 0x136668 (0xad24 bytes) 000479[0] 1786338240.371776 read 0x1c5e9f thru 0x1c6e3c (0xf9e bytes) 000480[0] 1786338240.415723 mapread 0x1b063d thru 0x1bcbcc (0xc590 bytes) 000481[0] 1786338240.556752 write 0x8b2c8e thru 0x8b71d2 (0x4545 bytes) 000482[0] 1786338240.649431 read 0x6899ae thru 0x68ff24 (0x6577 bytes) 000483[0] 1786338240.706087 mapwrite 0x5def83 thru 0x5e807b (0x90f9 bytes) 000484[0] 1786338240.907885 trunc from 0x8b71d3 to 0x67ca79 (0x23a759 bytes) 000485[0] 1786338240.949901 trunc from 0x67ca7a to 0x76548b (0xe8a12 bytes) 000488[0] 1786338241.016083 mapread 0x40aecd thru 0x40b14a (0x27e bytes) 000489[0] 1786338241.030850 mapread 0x52c23b thru 0x536b59 (0xa91f bytes) 000492[0] 1786338241.134060 write 0x3cef88 thru 0x3dd7f1 (0xe86a bytes) 000494[0] 1786338241.184738 mapread 0x05ead5 thru 0x06181c (0x2d48 bytes) 000498[1] 1786338241.222996 read 0x16c3b3 thru 0x16d17d (0xdcb bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8241 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000008[1] 1786338242.177564 write 0x988000 thru 0x988a62 (0xa63 bytes) 000009[0] 1786338242.203353 write_OD 0x1e4000 thru 0x1f0fff (0xd000 bytes) 000010[1] 1786338242.283036 mapwrite 0x99e000 thru 0x99e300 (0x301 bytes) truncating to largest ever: 0x534f0 000011[0] 1786338242.400088 trunc from 0x99e301 to 0x0534ef (0x94ae11 bytes) 000013[0] 1786338242.445633 mapread 0x00d000 thru 0x01149d (0x449e bytes) 000018[0] 1786338242.521960 mapread 0x034000 thru 0x0342b1 (0x2b2 bytes) truncating to largest ever: 0x7d8c62 000019[0] 1786338242.536447 trunc from 0x0534f0 to 0x7d8c61 (0x785772 bytes) 000020[0] 1786338242.586877 mapwrite 0x895000 thru 0x8a0b04 (0xbb05 bytes) 000021[1] 1786338242.797869 read 0x56b000 thru 0x57a133 (0xf134 bytes) 000022[0] 1786338242.851736 mapread 0x098000 thru 0x0a5549 (0xd54a bytes) 000025[0] 1786338242.905943 write_OD 0x8bc000 thru 0x8bdfff (0x2000 bytes) 000026[1] 1786338243.010132 write 0x34d000 thru 0x357b8b (0xab8c bytes) 000028[0] 1786338243.044075 read_OD 0x788000 thru 0x791fff (0xa000 bytes) 000030[1] 1786338243.136383 mapwrite 0x52c000 thru 0x539e76 (0xde77 bytes) 000035[1] 1786338243.409660 mapwrite 0x8a8000 thru 0x8b0440 (0x8441 bytes) 000039[0] 1786338243.690562 trunc from 0x8be000 to 0x2fcedf (0x5c1120 bytes) 000040[1] 1786338243.777015 mapwrite 0x7a6000 thru 0x7a7d77 (0x1d78 bytes) 000041[0] 1786338243.919888 write_OD 0x19d000 thru 0x19dfff (0x1000 bytes) 000042[0] 1786338243.989027 trunc from 0x7a7d78 to 0x327186 (0x480bf1 bytes) 000045[0] 1786338244.027862 write_OD 0x0c4000 thru 0x0c4fff (0x1000 bytes) 000046[0] 1786338244.088811 trunc from 0x327187 to 0x23e885 (0xe8901 bytes) 000050[0] 1786338244.174983 trunc from 0x23e886 to 0x06a8b5 (0x1d3fd0 bytes) 000051[1] 1786338244.220332 read 0x020000 thru 0x027221 (0x7222 bytes) 000052[1] 1786338244.303378 mapwrite 0x3f1000 thru 0x3fb3d1 (0xa3d2 bytes) 000053[1] 1786338244.701869 write 0x1bc000 thru 0x1cb6f0 (0xf6f1 bytes) 000054[1] 1786338244.739663 mapwrite 0x8db000 thru 0x8e9288 (0xe289 bytes) 000056[1] 1786338245.073477 trunc from 0x8e9289 to 0x31ed05 (0x5ca583 bytes) 000058[1] 1786338245.131946 mapread 0x1fd000 thru 0x20693b (0x993c bytes) 000059[0] 1786338245.134591 mapwrite 0x630000 thru 0x634e9d (0x4e9e bytes) 000061[0] 1786338245.293329 mapwrite 0x25b000 thru 0x266f92 (0xbf93 bytes) 000063[0] 1786338245.426134 read_OD 0x2d7000 thru 0x2e0fff (0xa000 bytes) 000067[0] 1786338245.463845 write_OD 0x4d4000 thru 0x4e1fff (0xe000 bytes) 000068[0] 1786338245.548886 trunc from 0x634e9e to 0x51861b (0x11c882 bytes) 000069[0] 1786338245.647564 mapread 0x205000 thru 0x20760c (0x260d bytes) truncating to largest ever: 0x7ee7d3 000071[0] 1786338245.693674 trunc from 0x51861c to 0x7ee7d2 (0x2d61b7 bytes) 000074[1] 1786338245.781701 write 0x2af000 thru 0x2b973d (0xa73e bytes) 000078[0] 1786338245.870096 read_OD 0x7bf000 thru 0x7c7fff (0x9000 bytes) 000079[0] 1786338245.917109 write_OD 0x643000 thru 0x646fff (0x4000 bytes) 000081[0] 1786338245.988972 trunc from 0x7ee7d3 to 0x0a51f6 (0x7495dc bytes) 000086[0] 1786338246.078147 trunc from 0x0a51f7 to 0x2f9f92 (0x254d9c bytes) 000087[1] 1786338246.129959 read 0x22c000 thru 0x2379be (0xb9bf bytes) 000088[0] 1786338246.251132 trunc from 0x2f9f93 to 0x31cdaf (0x22e1d bytes) 000089[1] 1786338246.360034 mapread 0x2f0000 thru 0x2f1227 (0x1228 bytes) 000090[1] 1786338246.412026 write 0x6b4000 thru 0x6bb036 (0x7037 bytes) truncating to largest ever: 0x9159bf 000091[0] 1786338246.463958 trunc from 0x6bb037 to 0x9159be (0x25a988 bytes) 000097[0] 1786338246.613129 trunc from 0x9159bf to 0x6a107a (0x274944 bytes) 000099[0] 1786338246.681673 mapread 0x363000 thru 0x36cade (0x9adf bytes) 000100[1] 1786338246.733696 trunc from 0x6a107b to 0x1f88e4 (0x4a8796 bytes) 000101[0] 1786338246.774125 trunc from 0x1f88e5 to 0x09ee88 (0x159a5c bytes) 000103[0] 1786338246.843707 write_OD 0x1af000 thru 0x1b1fff (0x3000 bytes) 000105[0] 1786338246.927645 read_OD 0x0f7000 thru 0x101fff (0xb000 bytes) 000106[0] 1786338246.940115 read_OD 0x0da000 thru 0x0e3fff (0xa000 bytes) 000109[0] 1786338246.955074 write_OD 0x964000 thru 0x965fff (0x2000 bytes) 000110[0] 1786338247.025868 trunc from 0x966000 to 0x1e8a34 (0x77d5cb bytes) 000112[0] 1786338247.084107 write_OD 0x436000 thru 0x43ffff (0xa000 bytes) 000113[1] 1786338247.164731 read 0x0f4000 thru 0x0fd0bd (0x90be bytes) 000114[1] 1786338247.176839 mapwrite 0x66d000 thru 0x67660d (0x960e bytes) 000116[1] 1786338247.359565 mapread 0x4ab000 thru 0x4affab (0x4fac bytes) 000118[0] 1786338247.425806 trunc from 0x67660e to 0x4cc216 (0x1aa3f7 bytes) 000122[0] 1786338247.552750 trunc from 0x4cc217 to 0x053ef8 (0x47831e bytes) 000124[0] 1786338247.612099 read_OD 0x02c000 thru 0x02efff (0x3000 bytes) 000126[0] 1786338247.629634 read_OD 0x032000 thru 0x035fff (0x4000 bytes) 000127[1] 1786338247.640926 write 0x5d6000 thru 0x5e2103 (0xc104 bytes) 000131[0] 1786338247.765637 mapwrite 0x45c000 thru 0x45cca3 (0xca4 bytes) 000133[0] 1786338247.992163 trunc from 0x5e2104 to 0x4ca933 (0x1177d0 bytes) 000135[1] 1786338248.065572 write 0x8a4000 thru 0x8adffc (0x9ffd bytes) 000137[1] 1786338248.111923 trunc from 0x8adffd to 0x4dad8e (0x3d326e bytes) 000138[0] 1786338248.171812 mapread 0x23a000 thru 0x2485b8 (0xe5b9 bytes) 000141[1] 1786338248.409058 mapread 0x37c000 thru 0x38a5f9 (0xe5fa bytes) 000143[0] 1786338248.450740 read_OD 0x3fa000 thru 0x401fff (0x8000 bytes) 000146[0] 1786338248.465238 trunc from 0x4dad8f to 0x2c0d0b (0x21a083 bytes) 000148[0] 1786338248.562605 trunc from 0x2c0d0c to 0x826a45 (0x565d3a bytes) 000149[1] 1786338248.617794 write 0x25c000 thru 0x25e7dc (0x27dd bytes) 000150[1] 1786338248.678534 mapread 0x7d7000 thru 0x7e1ac7 (0xaac8 bytes) 000153[0] 1786338248.749639 write_OD 0x2d3000 thru 0x2defff (0xc000 bytes) 000154[0] 1786338248.831853 write_OD 0x94a000 thru 0x94afff (0x1000 bytes) 000157[0] 1786338248.915024 write_OD 0x09a000 thru 0x09dfff (0x4000 bytes) 000161[0] 1786338248.982851 mapwrite 0x3a9000 thru 0x3a9c65 (0xc66 bytes) 000165[1] 1786338249.171537 read 0x575000 thru 0x58016c (0xb16d bytes) truncating to largest ever: 0x95aeb1 000166[0] 1786338249.223505 trunc from 0x94b000 to 0x95aeb0 (0xfeb1 bytes) 000167[1] 1786338249.299840 write 0x3ae000 thru 0x3b0251 (0x2252 bytes) 000168[0] 1786338249.357758 mapwrite 0x2e5000 thru 0x2e6262 (0x1263 bytes) 000171[0] 1786338249.650970 mapwrite 0x92d000 thru 0x930358 (0x3359 bytes) 000172[1] 1786338249.798351 read_OD 0x163000 thru 0x168fff (0x6000 bytes) 000173[1] 1786338249.870758 trunc from 0x95aeb1 to 0x8194ee (0x1419c2 bytes) 000174[1] 1786338249.952795 mapwrite 0x1d2000 thru 0x1e06ea (0xe6eb bytes) 000175[1] 1786338250.133840 mapwrite 0x278000 thru 0x2874cb (0xf4cc bytes) 000176[1] 1786338250.232957 mapwrite 0x824000 thru 0x831294 (0xd295 bytes) 000177[0] 1786338250.419704 read_OD 0x407000 thru 0x413fff (0xd000 bytes) 000178[0] 1786338250.478663 read_OD 0x541000 thru 0x54bfff (0xb000 bytes) 000179[1] 1786338250.514992 read_OD 0x09f000 thru 0x0a9fff (0xb000 bytes) 000180[0] 1786338250.539128 mapread 0x6a5000 thru 0x6b2afb (0xdafc bytes) 000181[0] 1786338250.577700 mapwrite 0x9ad000 thru 0x9aec7a (0x1c7b bytes) 000182[1] 1786338250.769085 trunc from 0x9aec7b to 0x738896 (0x2763e4 bytes) 000186[1] 1786338250.930978 mapread 0x2b3000 thru 0x2b72db (0x42dc bytes) 000187[0] 1786338250.960063 mapread 0x078000 thru 0x08584e (0xd84f bytes) 000192[1] 1786338251.026621 mapread 0x6ad000 thru 0x6b642e (0x942f bytes) 000195[1] 1786338251.051022 mapread 0x3b1000 thru 0x3b4d96 (0x3d97 bytes) 000197[1] 1786338251.078865 mapread 0x023000 thru 0x0269a1 (0x39a2 bytes) 000198[1] 1786338251.103739 mapwrite 0x3e7000 thru 0x3ee152 (0x7153 bytes) 000199[0] 1786338251.230638 read_OD 0x717000 thru 0x71efff (0x8000 bytes) 000201[0] 1786338251.277985 mapwrite 0x6c0000 thru 0x6c42b6 (0x42b7 bytes) 000202[0] 1786338251.422737 mapwrite 0x01c000 thru 0x02a933 (0xe934 bytes) 000203[0] 1786338251.680873 mapwrite 0x292000 thru 0x2a058a (0xe58b bytes) 000207[1] 1786338251.945474 write_OD 0x4f5000 thru 0x4f9fff (0x5000 bytes) 000208[0] 1786338252.050623 mapread 0x539000 thru 0x53b1a2 (0x21a3 bytes) 000209[0] 1786338252.102263 mapread 0x694000 thru 0x69c0d7 (0x80d8 bytes) 000210[1] 1786338252.133866 mapread 0x3b4000 thru 0x3b83f8 (0x43f9 bytes) 000212[0] 1786338252.147093 write_OD 0x1ed000 thru 0x1edfff (0x1000 bytes) 000213[1] 1786338252.220661 mapwrite 0x0ff000 thru 0x10b931 (0xc932 bytes) 000215[0] 1786338252.387123 mapwrite 0x42e000 thru 0x43b6f2 (0xd6f3 bytes) 000216[1] 1786338252.568429 read_OD 0x0fa000 thru 0x101fff (0x8000 bytes) 000220[0] 1786338252.654726 mapwrite 0x3b8000 thru 0x3baf68 (0x2f69 bytes) 000221[0] 1786338252.907611 read_OD 0x376000 thru 0x37cfff (0x7000 bytes) 000223[0] 1786338252.980557 read_OD 0x5ec000 thru 0x5f2fff (0x7000 bytes) 000224[1] 1786338252.998046 mapread 0x5b4000 thru 0x5bc33d (0x833e bytes) 000227[1] 1786338253.035861 mapread 0x33e000 thru 0x348479 (0xa47a bytes) 000228[0] 1786338253.059877 write_OD 0x2fb000 thru 0x301fff (0x7000 bytes) 000230[1] 1786338253.235509 mapwrite 0x002000 thru 0x008619 (0x661a bytes) 000232[0] 1786338253.463471 mapread 0x4eb000 thru 0x4ec2e3 (0x12e4 bytes) 000233[0] 1786338253.532118 trunc from 0x738897 to 0x642c9f (0xf5bf7 bytes) 000235[0] 1786338253.641674 write_OD 0x8ad000 thru 0x8b0fff (0x4000 bytes) 000237[0] 1786338253.739626 read_OD 0x4d9000 thru 0x4e6fff (0xe000 bytes) 000240[0] 1786338253.758661 mapwrite 0x0ce000 thru 0x0d1138 (0x3139 bytes) 000241[0] 1786338253.902395 mapread 0x25e000 thru 0x26c62e (0xe62f bytes) 000242[1] 1786338253.951005 read_OD 0x505000 thru 0x505fff (0x1000 bytes) 000244[0] 1786338253.981046 trunc from 0x8b1000 to 0x18a6fe (0x726901 bytes) truncating to largest ever: 0x9c0d80 000245[0] 1786338254.028735 trunc from 0x18a6ff to 0x9c0d7f (0x836681 bytes) 000248[0] 1786338254.096885 read_OD 0x2b9000 thru 0x2c5fff (0xd000 bytes) 000249[0] 1786338254.120473 write_OD 0x3f8000 thru 0x3f9fff (0x2000 bytes) 000250[1] 1786338254.183057 write_OD 0x213000 thru 0x21bfff (0x9000 bytes) 000253[0] 1786338254.260993 trunc from 0x9c0d80 to 0x3cb13d (0x5f5c42 bytes) 000254[0] 1786338254.340719 mapwrite 0x398000 thru 0x39a7bf (0x27c0 bytes) 000256[0] 1786338254.508087 mapwrite 0x01a000 thru 0x021d73 (0x7d74 bytes) 000257[0] 1786338254.639463 mapwrite 0x3c4000 thru 0x3cee60 (0xae61 bytes) 000261[0] 1786338254.838022 mapwrite 0x1ea000 thru 0x1f963d (0xf63e bytes) 000262[0] 1786338254.947595 write_OD 0x8a4000 thru 0x8aafff (0x7000 bytes) 000263[1] 1786338255.013776 read_OD 0x775000 thru 0x77afff (0x6000 bytes) 000264[0] 1786338255.026751 trunc from 0x8ab000 to 0x0830fc (0x827f03 bytes) 000270[0] 1786338255.147959 write_OD 0x86f000 thru 0x877fff (0x9000 bytes) 000271[1] 1786338255.240059 mapwrite 0x0f3000 thru 0x0fb141 (0x8142 bytes) 000272[0] 1786338255.494301 mapwrite 0x813000 thru 0x8192a1 (0x62a2 bytes) 000275[0] 1786338255.778578 mapwrite 0x083000 thru 0x08fe77 (0xce78 bytes) 000277[0] 1786338255.916722 write_OD 0x726000 thru 0x731fff (0xc000 bytes) 000278[1] 1786338256.001943 write_OD 0x60b000 thru 0x611fff (0x7000 bytes) 000281[0] 1786338256.081728 read_OD 0x6b1000 thru 0x6b1fff (0x1000 bytes) 000282[1] 1786338256.091106 read_OD 0x368000 thru 0x36efff (0x7000 bytes) 000283[0] 1786338256.098721 mapread 0x614000 thru 0x621a83 (0xda84 bytes) 000284[1] 1786338256.136787 trunc from 0x878000 to 0x1d912e (0x69eed1 bytes) 000285[0] 1786338256.203899 write_OD 0x61d000 thru 0x629fff (0xd000 bytes) 000289[0] 1786338256.281852 read_OD 0x48f000 thru 0x49cfff (0xe000 bytes) 000291[0] 1786338256.291923 trunc from 0x62a000 to 0x002b3d (0x6274c2 bytes) 000294[0] 1786338256.358709 trunc from 0x002b3e to 0x0a3073 (0xa0536 bytes) 000295[0] 1786338256.391601 write_OD 0x1e7000 thru 0x1edfff (0x7000 bytes) 000297[0] 1786338256.473431 write_OD 0x89a000 thru 0x8a0fff (0x7000 bytes) 000298[1] 1786338256.565515 write_OD 0x131000 thru 0x137fff (0x7000 bytes) 000300[1] 1786338256.677197 trunc from 0x8a1000 to 0x6c000a (0x1e0ff5 bytes) 000302[0] 1786338256.773964 write_OD 0x62b000 thru 0x636fff (0xc000 bytes) 000304[0] 1786338256.868253 write_OD 0x3ac000 thru 0x3b4fff (0x9000 bytes) 000305[1] 1786338256.972393 read_OD 0x19f000 thru 0x1adfff (0xf000 bytes) 000307[1] 1786338256.994914 mapread 0x199000 thru 0x19abe3 (0x1be4 bytes) 000308[1] 1786338257.013480 write_OD 0x09a000 thru 0x0a0fff (0x7000 bytes) 000315[0] 1786338257.122907 trunc from 0x6c000b to 0x8f15e4 (0x2315da bytes) 000316[0] 1786338257.176768 write_OD 0x518000 thru 0x519fff (0x2000 bytes) 000317[0] 1786338257.244808 mapread 0x60a000 thru 0x60a8f9 (0x8fa bytes) 000320[1] 1786338257.263698 mapread 0x427000 thru 0x42a8c0 (0x38c1 bytes) 000321[1] 1786338257.313020 write_OD 0x74d000 thru 0x74efff (0x2000 bytes) 000322[1] 1786338257.404671 mapread 0x00d000 thru 0x012652 (0x5653 bytes) 000329[0] 1786338257.457702 mapread 0x4e7000 thru 0x4f60f6 (0xf0f7 bytes) truncating to largest ever: 0x9cae7c 000330[1] 1786338257.497631 trunc from 0x8f15e5 to 0x9cae7b (0xd9897 bytes) 000331[0] 1786338257.552293 write_OD 0x513000 thru 0x518fff (0x6000 bytes) 000332[1] 1786338257.623087 trunc from 0x9cae7c to 0x3f589f (0x5d55dc bytes) 000337[1] 1786338257.712755 write_OD 0x9d5000 thru 0x9e0fff (0xc000 bytes) 000339[1] 1786338257.774643 write_OD 0x2fb000 thru 0x300fff (0x6000 bytes) 000340[1] 1786338257.839782 mapwrite 0x384000 thru 0x385c90 (0x1c91 bytes) 000341[1] 1786338257.924369 trunc from 0x9e1000 to 0x75a8b4 (0x28674b bytes) 000344[1] 1786338258.001089 write_OD 0x63d000 thru 0x64bfff (0xf000 bytes) 000346[1] 1786338258.069836 read_OD 0x54e000 thru 0x552fff (0x5000 bytes) 000347[1] 1786338258.080671 trunc from 0x75a8b5 to 0x3397bc (0x4210f8 bytes) 000349[1] 1786338258.123395 read_OD 0x1ee000 thru 0x1eefff (0x1000 bytes) 000350[1] 1786338258.146850 mapread 0x269000 thru 0x26d2a3 (0x42a4 bytes) 000356[1] 1786338258.243904 write_OD 0x9e7000 thru 0x9ebfff (0x5000 bytes) 000357[0] 1786338258.318519 mapread 0x808000 thru 0x81480d (0xc80e bytes) 000359[1] 1786338258.369455 mapwrite 0x1e9000 thru 0x1f867a (0xf67b bytes) 000361[1] 1786338258.657672 mapread 0x32a000 thru 0x32d2e8 (0x32e9 bytes) 000362[0] 1786338258.681969 read_OD 0x71a000 thru 0x71bfff (0x2000 bytes) 000365[1] 1786338258.723735 trunc from 0x9ec000 to 0x0e693b (0x9056c4 bytes) 000366[0] 1786338258.764392 mapwrite 0x5f3000 thru 0x5fcc6a (0x9c6b bytes) 000367[0] 1786338258.979647 trunc from 0x5fcc6b to 0x331094 (0x2cbbd6 bytes) 000372[0] 1786338259.108580 write_OD 0x627000 thru 0x630fff (0xa000 bytes) 000373[0] 1786338259.174848 write_OD 0x7d2000 thru 0x7dafff (0x9000 bytes) 000376[0] 1786338259.237885 write_OD 0x27e000 thru 0x28bfff (0xe000 bytes) 000377[1] 1786338259.326961 mapread 0x7a4000 thru 0x7b352e (0xf52f bytes) 000379[0] 1786338259.486146 trunc from 0x7db000 to 0x4a92c5 (0x331d3a bytes) 000381[0] 1786338259.586715 trunc from 0x4a92c6 to 0x66f6eb (0x1c6426 bytes) 000383[0] 1786338259.630033 read_OD 0x303000 thru 0x30bfff (0x9000 bytes) 000386[0] 1786338259.670911 read_OD 0x390000 thru 0x39cfff (0xd000 bytes) 000387[1] 1786338259.688904 mapread 0x0ec000 thru 0x0f0238 (0x4239 bytes) 000390[0] 1786338259.719472 write_OD 0x01e000 thru 0x023fff (0x6000 bytes) 000392[1] 1786338259.793603 mapread 0x00d000 thru 0x00e236 (0x1237 bytes) 000393[0] 1786338259.812256 write_OD 0x88a000 thru 0x88afff (0x1000 bytes) 000394[0] 1786338259.882064 trunc from 0x88b000 to 0x2fe220 (0x58cddf bytes) 000396[1] 1786338259.937805 trunc from 0x2fe221 to 0x13c6d6 (0x1c1b4a bytes) 000398[1] 1786338260.023963 read_OD 0x047000 thru 0x055fff (0xf000 bytes) 000400[1] 1786338260.061803 write_OD 0x940000 thru 0x94dfff (0xe000 bytes) 000403[0] 1786338260.219018 read_OD 0x2a4000 thru 0x2abfff (0x8000 bytes) 000404[0] 1786338260.233082 trunc from 0x94e000 to 0x178342 (0x7d5cbd bytes) 000405[1] 1786338260.302592 read_OD 0x0c2000 thru 0x0cbfff (0xa000 bytes) 000406[1] 1786338260.362890 mapwrite 0x3d8000 thru 0x3e582e (0xd82f bytes) 000407[0] 1786338260.688836 mapread 0x12c000 thru 0x1397f8 (0xd7f9 bytes) 000409[0] 1786338260.803821 write_OD 0x2e9000 thru 0x2ecfff (0x4000 bytes) 000412[0] 1786338260.878595 mapwrite 0x166000 thru 0x1673d9 (0x13da bytes) 000413[0] 1786338260.974656 trunc from 0x3e582f to 0x4cf942 (0xea114 bytes) 000415[1] 1786338261.038537 write_OD 0x03a000 thru 0x041fff (0x8000 bytes) 000419[1] 1786338261.145453 trunc from 0x4cf943 to 0x6d323f (0x2038fd bytes) 000424[0] 1786338261.284745 mapwrite 0x912000 thru 0x916126 (0x4127 bytes) 000425[1] 1786338261.459913 write_OD 0x7e3000 thru 0x7f1fff (0xf000 bytes) 000428[0] 1786338261.591359 mapread 0x259000 thru 0x25dda2 (0x4da3 bytes) 000430[0] 1786338261.634542 read_OD 0x6c8000 thru 0x6d4fff (0xd000 bytes) 000432[0] 1786338261.649286 mapread 0x336000 thru 0x345aa5 (0xfaa6 bytes) 000433[0] 1786338261.685495 write_OD 0x4b3000 thru 0x4b6fff (0x4000 bytes) 000434[0] 1786338261.748539 mapread 0x7c5000 thru 0x7cd508 (0x8509 bytes) 000436[1] 1786338261.781766 mapwrite 0x444000 thru 0x44df9e (0x9f9f bytes) 000441[1] 1786338262.031657 write_OD 0x788000 thru 0x794fff (0xd000 bytes) 000444[1] 1786338262.099225 write_OD 0x508000 thru 0x513fff (0xc000 bytes) 000449[1] 1786338262.165144 trunc from 0x916127 to 0x8886aa (0x8da7c bytes) 000451[0] 1786338262.235743 write_OD 0x007000 thru 0x00efff (0x8000 bytes) 000452[1] 1786338262.333961 write_OD 0x4b7000 thru 0x4c3fff (0xd000 bytes) 000453[0] 1786338262.413840 trunc from 0x8886ab to 0x4c09d6 (0x3c7cd4 bytes) 000454[0] 1786338262.491015 mapread 0x490000 thru 0x499f09 (0x9f0a bytes) 000455[0] 1786338262.524016 write_OD 0x860000 thru 0x860fff (0x1000 bytes) 000456[0] 1786338262.592326 trunc from 0x861000 to 0x340cff (0x520300 bytes) 000457[0] 1786338262.637161 read_OD 0x0ee000 thru 0x0f8fff (0xb000 bytes) 000458[0] 1786338262.670954 trunc from 0x340d00 to 0x90a797 (0x5c9a98 bytes) 000459[0] 1786338262.751580 write_OD 0x62c000 thru 0x62cfff (0x1000 bytes) 000460[1] 1786338262.829154 mapread 0x09a000 thru 0x0a548c (0xb48d bytes) 000461[0] 1786338262.941011 read_OD 0x2f4000 thru 0x300fff (0xd000 bytes) 000467[0] 1786338262.955828 mapwrite 0x983000 thru 0x98f1ce (0xc1cf bytes) 000468[1] 1786338263.199905 write_OD 0x22a000 thru 0x231fff (0x8000 bytes) 000469[0] 1786338263.278037 mapread 0x231000 thru 0x231e1a (0xe1b bytes) 000470[1] 1786338263.297811 mapwrite 0x009000 thru 0x015f5f (0xcf60 bytes) 000471[1] 1786338263.598756 read_OD 0x69a000 thru 0x6a0fff (0x7000 bytes) 000472[1] 1786338263.622539 write_OD 0x410000 thru 0x41bfff (0xc000 bytes) 000475[1] 1786338263.707789 mapread 0x2f1000 thru 0x2f93ff (0x8400 bytes) 000477[1] 1786338263.732240 mapread 0x3f1000 thru 0x400a5f (0xfa60 bytes) 000478[1] 1786338263.766071 read_OD 0x10e000 thru 0x110fff (0x3000 bytes) 000480[0] 1786338263.776704 read_OD 0x12f000 thru 0x131fff (0x3000 bytes) 000482[0] 1786338263.785329 mapread 0x5ad000 thru 0x5b0224 (0x3225 bytes) 000484[1] 1786338263.817961 mapwrite 0x9f9000 thru 0x9ff2c2 (0x62c3 bytes) 000487[1] 1786338264.023049 trunc from 0x9ff2c3 to 0x7fa268 (0x20505a bytes) 000488[0] 1786338264.070964 mapwrite 0x252000 thru 0x25b189 (0x918a bytes) 000489[1] 1786338264.204595 read_OD 0x109000 thru 0x10dfff (0x5000 bytes) 000490[1] 1786338264.254394 read_OD 0x2be000 thru 0x2c7fff (0xa000 bytes) 000494[1] 1786338264.288665 mapwrite 0x606000 thru 0x614412 (0xe413 bytes) 000497[1] 1786338264.483692 trunc from 0x7fa269 to 0x839a95 (0x3f82d bytes) 000498[0] 1786338264.531552 write_OD 0x667000 thru 0x668fff (0x2000 bytes) 000500[1] 1786338264.603222 mapwrite 0x4c0000 thru 0x4c455b (0x455c bytes) All operations completed A-OK! PASS 16a (60s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:04:33 (1786338273) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.435427 s, 19.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.412355 s, 20.3 MB/s Chance of close/open is 1 in 50 Seed set to 8277 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1786338278.089953 mapwrite 0x0914d3 thru 0x0929a6 (0x14d4 bytes) 000002[0] 1786338278.170359 write 0x58c117 thru 0x58cd0c (0xbf6 bytes) 000004[0] 1786338278.216495 write 0x6fedd1 thru 0x6fee06 (0x36 bytes) 000005[0] 1786338278.219110 write 0x7da8d3 thru 0x7daa30 (0x15e bytes) 000008[1] 1786338278.257470 write 0x9f0610 thru 0x9f15ae (0xf9f bytes) truncating to largest ever: 0x6df50e 000012[0] 1786338278.359691 trunc from 0x9f15af to 0x6df50d (0x3120a1 bytes) 000013[1] 1786338278.407082 read 0x500466 thru 0x5015a0 (0x113b bytes) 000014[0] 1786338278.440481 mapwrite 0x3a3aad thru 0x3a4c44 (0x1198 bytes) 000017[0] 1786338278.567014 write 0x66dcfb thru 0x66e514 (0x81a bytes) 000018[1] 1786338278.646807 mapread 0x3d0f87 thru 0x3d2967 (0x19e1 bytes) 000019[0] 1786338278.733611 read 0x41e4ef thru 0x41fcdb (0x17ed bytes) 000021[0] 1786338278.783765 mapwrite 0x4af9d2 thru 0x4b1574 (0x1ba3 bytes) 000023[0] 1786338278.922149 mapread 0x3e69b6 thru 0x3e86fb (0x1d46 bytes) 000024[1] 1786338278.940573 mapread 0x505278 thru 0x5064e6 (0x126f bytes) 000030[1] 1786338278.992538 trunc from 0x6df50e to 0x620a8a (0xbea83 bytes) 000032[0] 1786338279.095428 mapread 0x1857b2 thru 0x185ef2 (0x741 bytes) 000033[0] 1786338279.132666 trunc from 0x620a8b to 0x1a75b5 (0x4794d5 bytes) 000034[0] 1786338279.226017 mapwrite 0x70fb18 thru 0x70fbf5 (0xde bytes) 000035[1] 1786338279.334959 mapread 0x70703f thru 0x7070e6 (0xa8 bytes) 000036[0] 1786338279.409005 mapread 0x37141d thru 0x372f96 (0x1b7a bytes) 000037[1] 1786338279.455559 write 0x7364bf thru 0x73763f (0x1181 bytes) 000038[0] 1786338279.519928 write 0x631c66 thru 0x63276d (0xb08 bytes) 000039[0] 1786338279.556073 mapread 0x6535cb thru 0x653d95 (0x7cb bytes) 000040[0] 1786338279.563895 mapwrite 0x1a399b thru 0x1a41d6 (0x83c bytes) 000043[0] 1786338279.688543 read 0x4cf61b thru 0x4cf947 (0x32d bytes) 000044[1] 1786338279.696518 mapwrite 0x231291 thru 0x2324a2 (0x1212 bytes) truncating to largest ever: 0x943944 000046[0] 1786338279.912507 trunc from 0x737640 to 0x943943 (0x20c304 bytes) 000051[0] 1786338280.052620 trunc from 0x943944 to 0x86b628 (0xd831b bytes) 000052[1] 1786338280.110902 read 0x25d044 thru 0x25e0f7 (0x10b4 bytes) 000053[1] 1786338280.173634 read 0x3d4804 thru 0x3d62e7 (0x1ae4 bytes) 000054[1] 1786338280.185504 write 0x746645 thru 0x7472c8 (0xc84 bytes) 000055[1] 1786338280.240586 read 0x7f83a1 thru 0x7f8ca9 (0x909 bytes) 000056[0] 1786338280.272998 write 0x37fa7a thru 0x37fdee (0x375 bytes) 000058[1] 1786338280.368986 mapread 0x4778c1 thru 0x478f12 (0x1652 bytes) 000060[0] 1786338280.449890 mapread 0x127c64 thru 0x12923e (0x15db bytes) 000062[0] 1786338280.532875 trunc from 0x86b629 to 0x5950fa (0x2d652e bytes) 000067[1] 1786338280.728864 mapread 0x005183 thru 0x0068e2 (0x1760 bytes) 000068[1] 1786338280.782318 write 0x5c9f19 thru 0x5cae94 (0xf7c bytes) 000069[1] 1786338280.801620 mapwrite 0x42fc48 thru 0x4301b9 (0x572 bytes) 000070[1] 1786338280.858236 mapwrite 0x7f5e4a thru 0x7f662b (0x7e2 bytes) 000072[1] 1786338281.014154 read 0x15b779 thru 0x15d4ca (0x1d52 bytes) 000076[0] 1786338281.059821 mapwrite 0x243942 thru 0x243bdd (0x29c bytes) 000078[0] 1786338281.192967 mapread 0x10a51d thru 0x10c281 (0x1d65 bytes) 000079[1] 1786338281.257336 write 0x22f2bd thru 0x230a65 (0x17a9 bytes) 000081[1] 1786338281.292486 mapread 0x5f3294 thru 0x5f380c (0x579 bytes) 000083[0] 1786338281.364991 trunc from 0x7f662c to 0x11ff08 (0x6d6723 bytes) 000084[0] 1786338281.462517 read 0x0cea6f thru 0x0cf69a (0xc2c bytes) 000085[0] 1786338281.508828 mapread 0x082b79 thru 0x083ad3 (0xf5b bytes) 000089[0] 1786338281.547049 mapread 0x01ebc9 thru 0x02031f (0x1757 bytes) 000091[0] 1786338281.601123 trunc from 0x11ff09 to 0x06f31e (0xb0bea bytes) 000093[0] 1786338281.733527 read 0x050e62 thru 0x052dbe (0x1f5d bytes) 000095[1] 1786338281.759964 read 0x05b9ed thru 0x05c3f8 (0xa0c bytes) 000096[1] 1786338281.824882 read 0x00ea79 thru 0x00f086 (0x60e bytes) 000098[0] 1786338281.852068 write 0x780a7c thru 0x780e83 (0x408 bytes) 000101[1] 1786338281.898446 read 0x686eca thru 0x6870c9 (0x200 bytes) 000102[1] 1786338281.928164 mapwrite 0x1b8eef thru 0x1b9f62 (0x1074 bytes) 000103[1] 1786338282.028045 write 0x064533 thru 0x065f28 (0x19f6 bytes) 000104[0] 1786338282.044724 mapread 0x4b2301 thru 0x4b31fa (0xefa bytes) 000105[1] 1786338282.067832 read 0x224be8 thru 0x22527e (0x697 bytes) 000108[0] 1786338282.114731 read 0x5a78a2 thru 0x5a8833 (0xf92 bytes) 000109[0] 1786338282.135164 read 0x375361 thru 0x37607b (0xd1b bytes) 000111[1] 1786338282.177924 write 0x7241bd thru 0x7257ea (0x162e bytes) 000112[0] 1786338282.225327 mapread 0x384768 thru 0x384e47 (0x6e0 bytes) 000114[0] 1786338282.258807 trunc from 0x780e84 to 0x3bcb6e (0x3c4315 bytes) 000116[0] 1786338282.428857 trunc from 0x3bcb6f to 0x061f32 (0x35ac3c bytes) 000119[1] 1786338282.544026 read 0x01ccfc thru 0x01d6e3 (0x9e8 bytes) 000121[0] 1786338282.623526 trunc from 0x061f33 to 0x1caea7 (0x168f75 bytes) 000122[1] 1786338282.713827 mapwrite 0x2b4ac9 thru 0x2b57bf (0xcf7 bytes) 000123[0] 1786338282.906470 read 0x012600 thru 0x01327c (0xc7d bytes) 000124[0] 1786338283.037985 write 0x9a3f72 thru 0x9a4f41 (0xfd0 bytes) 000125[0] 1786338283.089107 write 0x46ed62 thru 0x46f071 (0x310 bytes) 000126[1] 1786338283.219841 mapread 0x7c6234 thru 0x7c6854 (0x621 bytes) 000127[0] 1786338283.310106 read 0x38812d thru 0x388c9a (0xb6e bytes) 000131[0] 1786338283.378308 read 0x2cf257 thru 0x2cf757 (0x501 bytes) 000132[1] 1786338283.379345 trunc from 0x9a4f42 to 0x72c249 (0x278cf8 bytes) 000134[1] 1786338283.457094 write 0x94c887 thru 0x94d23d (0x9b7 bytes) 000135[1] 1786338283.463821 mapread 0x70df37 thru 0x70f568 (0x1632 bytes) 000140[1] 1786338283.797931 trunc from 0x94d23e to 0x721725 (0x22bb18 bytes) 000141[1] 1786338283.824737 mapread 0x66a218 thru 0x66b9bb (0x17a4 bytes) 000142[1] 1786338283.855764 mapwrite 0x679cc1 thru 0x67a6bd (0x9fd bytes) 000143[1] 1786338283.929137 mapread 0x11028e thru 0x110976 (0x6e9 bytes) 000148[0] 1786338284.012150 mapwrite 0x65a584 thru 0x65ab9d (0x61a bytes) 000149[1] 1786338284.133990 mapread 0x0db04c thru 0x0dce93 (0x1e48 bytes) 000150[1] 1786338284.195417 mapwrite 0x3b0f71 thru 0x3b2124 (0x11b4 bytes) 000151[1] 1786338284.300060 trunc from 0x721726 to 0x8fabf3 (0x1d94ce bytes) 000152[0] 1786338284.344712 trunc from 0x8fabf4 to 0x36c27b (0x58e978 bytes) 000154[1] 1786338284.421138 mapwrite 0x354fc3 thru 0x35688a (0x18c8 bytes) 000155[1] 1786338284.517963 mapread 0x1ff45e thru 0x200e7f (0x1a22 bytes) truncating to largest ever: 0x97e2fe 000157[1] 1786338284.577007 trunc from 0x36c27c to 0x97e2fd (0x612082 bytes) 000159[1] 1786338284.661607 write 0x3a48c9 thru 0x3a6689 (0x1dc1 bytes) 000165[0] 1786338284.772417 read 0x679d36 thru 0x67af36 (0x1201 bytes) 000166[0] 1786338284.814251 mapwrite 0x188816 thru 0x189a3c (0x1227 bytes) 000167[1] 1786338284.943693 mapwrite 0x743390 thru 0x744b9f (0x1810 bytes) 000168[1] 1786338285.084998 mapwrite 0x529c52 thru 0x52ad78 (0x1127 bytes) 000169[0] 1786338285.278617 mapread 0x04d8fd thru 0x04e74b (0xe4f bytes) 000170[0] 1786338285.336706 mapread 0x5142e1 thru 0x514b5a (0x87a bytes) 000171[1] 1786338285.367152 read 0x2c2d1e thru 0x2c43e4 (0x16c7 bytes) 000175[1] 1786338285.403523 write 0x1550fd thru 0x156359 (0x125d bytes) 000177[1] 1786338285.426877 write 0x792518 thru 0x7943cd (0x1eb6 bytes) 000178[1] 1786338285.458093 write 0x36bb89 thru 0x36cbf5 (0x106d bytes) 000179[1] 1786338285.470121 write 0x2a6f6a thru 0x2a7eca (0xf61 bytes) 000180[1] 1786338285.480041 trunc from 0x97e2fe to 0x62316a (0x35b193 bytes) 000181[0] 1786338285.546092 write 0x8f3ed8 thru 0x8f3f01 (0x2a bytes) 000184[1] 1786338285.647818 mapwrite 0x9a7c4c thru 0x9a86ba (0xa6f bytes) 000187[1] 1786338285.759033 write 0x849230 thru 0x84a180 (0xf51 bytes) 000190[0] 1786338285.842800 read 0x7a0414 thru 0x7a186b (0x1458 bytes) 000191[1] 1786338285.896776 mapread 0x8b5c02 thru 0x8b6adb (0xeda bytes) 000193[0] 1786338285.932927 read 0x76b22b thru 0x76b9c3 (0x799 bytes) 000194[0] 1786338285.959888 trunc from 0x9a86bb to 0x0572e0 (0x9513da bytes) 000195[1] 1786338286.086777 mapwrite 0x7926e6 thru 0x793d3a (0x1655 bytes) 000196[0] 1786338286.265732 write 0x67d072 thru 0x67ef3f (0x1ece bytes) 000199[1] 1786338286.393705 write 0x4e66e4 thru 0x4e83de (0x1cfb bytes) 000200[0] 1786338286.503888 write 0x8d9fd2 thru 0x8db055 (0x1084 bytes) 000202[0] 1786338286.647863 trunc from 0x8db056 to 0x8ea15e (0xf109 bytes) 000205[1] 1786338286.819856 mapread 0x1606fb thru 0x160c3a (0x540 bytes) 000206[1] 1786338286.887610 trunc from 0x8ea15f to 0x0e6da8 (0x8033b6 bytes) 000207[1] 1786338287.016523 trunc from 0x0e6da9 to 0x04b5b6 (0x9b7f2 bytes) 000208[0] 1786338287.090971 write 0x6669c2 thru 0x668998 (0x1fd7 bytes) 000209[1] 1786338287.156784 read 0x3821ad thru 0x383f9e (0x1df2 bytes) 000210[0] 1786338287.199052 read 0x27a018 thru 0x27abfd (0xbe6 bytes) 000212[0] 1786338287.249795 trunc from 0x668999 to 0x24a7d7 (0x41e1c1 bytes) 000213[0] 1786338287.309028 mapread 0x1c29bc thru 0x1c3a4f (0x1094 bytes) 000214[0] 1786338287.349097 trunc from 0x24a7d8 to 0x94be52 (0x70167b bytes) 000216[1] 1786338287.444058 trunc from 0x94be53 to 0x777a04 (0x1d444e bytes) 000217[0] 1786338287.534557 trunc from 0x777a05 to 0x35ae8c (0x41cb78 bytes) 000218[0] 1786338287.622080 mapwrite 0x0c182b thru 0x0c2335 (0xb0b bytes) 000219[1] 1786338287.698710 read 0x30123b thru 0x302ab2 (0x1878 bytes) 000221[0] 1786338287.845107 write 0x1b2f56 thru 0x1b3bb9 (0xc64 bytes) 000223[1] 1786338287.915997 mapwrite 0x2b025b thru 0x2b180d (0x15b3 bytes) 000224[0] 1786338288.042764 write 0x618b15 thru 0x61a20b (0x16f7 bytes) 000225[0] 1786338288.073470 write 0x72e35d thru 0x72f72e (0x13d2 bytes) 000226[1] 1786338288.096701 mapwrite 0x4b202a thru 0x4b3d9e (0x1d75 bytes) 000229[0] 1786338288.227738 mapwrite 0x6f2c7d thru 0x6f3005 (0x389 bytes) 000231[0] 1786338288.347746 trunc from 0x72f72f to 0x580872 (0x1aeebc bytes) 000234[1] 1786338288.475824 mapwrite 0x2d0641 thru 0x2d1e28 (0x17e8 bytes) 000235[0] 1786338288.681716 write 0x9bac73 thru 0x9bba6c (0xdfa bytes) 000236[1] 1786338288.771885 read 0x877dae thru 0x878568 (0x7bb bytes) 000238[1] 1786338288.836673 read 0x2655bb thru 0x267292 (0x1cd8 bytes) 000240[1] 1786338288.862116 mapread 0x8ae92b thru 0x8af48d (0xb63 bytes) 000241[0] 1786338288.900894 mapread 0x1b679a thru 0x1b70bd (0x924 bytes) 000245[1] 1786338288.936932 mapread 0x1960bb thru 0x196c49 (0xb8f bytes) 000246[1] 1786338288.954892 write 0x935e58 thru 0x935f7f (0x128 bytes) 000249[0] 1786338289.040858 read 0x9502e7 thru 0x951f10 (0x1c2a bytes) 000250[0] 1786338289.098578 read 0x1c22b6 thru 0x1c2de1 (0xb2c bytes) 000252[0] 1786338289.126443 mapwrite 0x3b0cd4 thru 0x3b18b7 (0xbe4 bytes) 000253[0] 1786338289.226990 mapread 0x1e273f thru 0x1e2a86 (0x348 bytes) 000255[0] 1786338289.262689 read 0x3e6c5e thru 0x3e8158 (0x14fb bytes) 000257[1] 1786338289.271693 mapwrite 0x29ffe0 thru 0x2a1de8 (0x1e09 bytes) 000258[0] 1786338289.415897 read 0x09806e thru 0x098509 (0x49c bytes) 000260[0] 1786338289.466706 write 0x299deb thru 0x29af76 (0x118c bytes) 000263[1] 1786338289.498810 trunc from 0x9bba6d to 0x71f294 (0x29c7d8 bytes) 000264[0] 1786338289.593836 trunc from 0x71f295 to 0x254496 (0x4cadfe bytes) 000267[0] 1786338289.789014 mapwrite 0x5e95ec thru 0x5ea07b (0xa90 bytes) 000269[0] 1786338289.900099 mapwrite 0x5a0859 thru 0x5a23d5 (0x1b7d bytes) 000271[0] 1786338290.001844 write 0x5e1450 thru 0x5e16d9 (0x28a bytes) 000275[1] 1786338290.044544 mapwrite 0x03f7f4 thru 0x040203 (0xa10 bytes) 000279[0] 1786338290.330774 read 0x2c3671 thru 0x2c4073 (0xa03 bytes) 000283[0] 1786338290.372303 trunc from 0x5ea07c to 0x532f14 (0xb7167 bytes) truncating to largest ever: 0x97f9c1 000284[1] 1786338290.437868 trunc from 0x532f15 to 0x97f9c0 (0x44caac bytes) 000287[1] 1786338290.541886 mapwrite 0x852ea5 thru 0x854499 (0x15f5 bytes) 000288[0] 1786338290.611322 trunc from 0x97f9c1 to 0x879e65 (0x105b5b bytes) 000292[1] 1786338290.766736 mapread 0x638007 thru 0x6381c5 (0x1bf bytes) 000293[1] 1786338290.811302 mapwrite 0x798647 thru 0x79941c (0xdd6 bytes) 000294[0] 1786338290.917794 mapread 0x132b45 thru 0x1330fa (0x5b6 bytes) 000296[0] 1786338291.005807 mapread 0x2dd190 thru 0x2dd50b (0x37c bytes) 000297[0] 1786338291.043644 mapwrite 0x1b1361 thru 0x1b25d7 (0x1277 bytes) 000300[1] 1786338291.166732 read 0x777615 thru 0x778d45 (0x1731 bytes) 000302[1] 1786338291.219141 mapwrite 0x897a49 thru 0x897e82 (0x43a bytes) 000303[1] 1786338291.325151 mapread 0x503caa thru 0x505999 (0x1cf0 bytes) 000305[1] 1786338291.395701 write 0x9e3857 thru 0x9e4fd1 (0x177b bytes) 000309[0] 1786338291.436149 trunc from 0x9e4fd2 to 0x8d7c6d (0x10d364 bytes) 000312[0] 1786338291.530733 mapread 0x84725f thru 0x8476ca (0x46c bytes) 000314[1] 1786338291.590589 mapread 0x17470b thru 0x175cbb (0x15b1 bytes) 000318[1] 1786338291.721570 mapread 0x686389 thru 0x6866fd (0x375 bytes) 000319[0] 1786338291.760779 mapread 0x41a1bb thru 0x41bd2d (0x1b73 bytes) 000321[1] 1786338291.773854 trunc from 0x8d7c6e to 0x20bb95 (0x6cc0d8 bytes) 000322[0] 1786338291.873951 write 0x14209d thru 0x1426dc (0x640 bytes) 000327[1] 1786338292.023649 mapread 0x1a090a thru 0x1a277e (0x1e75 bytes) 000328[0] 1786338292.093081 mapwrite 0x533ad4 thru 0x533fd1 (0x4fe bytes) 000330[1] 1786338292.356160 mapread 0x2d0d88 thru 0x2d1503 (0x77c bytes) 000332[1] 1786338292.528677 trunc from 0x533fd2 to 0x000f83 (0x53304e bytes) 000334[1] 1786338292.677921 mapwrite 0x5d42e0 thru 0x5d5ad5 (0x17f6 bytes) 000335[1] 1786338292.854925 mapwrite 0x5ab1b1 thru 0x5ab4a3 (0x2f3 bytes) 000337[1] 1786338292.996371 read 0x1a51a8 thru 0x1a682e (0x1687 bytes) 000338[0] 1786338293.007734 write 0x0945a0 thru 0x096455 (0x1eb6 bytes) 000339[1] 1786338293.067950 write 0x606234 thru 0x6075b2 (0x137f bytes) 000341[1] 1786338293.125716 read 0x06a924 thru 0x06c73b (0x1e18 bytes) 000342[0] 1786338293.184039 mapwrite 0x396cef thru 0x398c6f (0x1f81 bytes) 000343[0] 1786338293.382220 write 0x564681 thru 0x565c72 (0x15f2 bytes) 000344[0] 1786338293.475258 trunc from 0x6075b3 to 0x684d14 (0x7d762 bytes) 000346[0] 1786338293.578956 mapread 0x22971a thru 0x229780 (0x67 bytes) 000349[0] 1786338293.703456 read 0x4d2a88 thru 0x4d2e4f (0x3c8 bytes) 000352[0] 1786338293.792064 mapread 0x608754 thru 0x60a712 (0x1fbf bytes) 000353[1] 1786338293.827741 mapwrite 0x8f3026 thru 0x8f4bc9 (0x1ba4 bytes) 000355[1] 1786338294.091930 read 0x3c0f27 thru 0x3c1213 (0x2ed bytes) 000357[1] 1786338294.186735 mapread 0x5872c4 thru 0x588272 (0xfaf bytes) 000358[0] 1786338294.206549 trunc from 0x8f4bca to 0x239cb0 (0x6baf19 bytes) 000359[0] 1786338294.295020 trunc from 0x239cb1 to 0x50e0ed (0x2d443d bytes) 000360[0] 1786338294.357860 mapwrite 0x3f2460 thru 0x3f348e (0x102f bytes) 000364[1] 1786338294.539683 mapwrite 0x526e6a thru 0x5285c7 (0x175e bytes) 000366[1] 1786338294.811421 trunc from 0x5285c8 to 0x7bfd36 (0x29776f bytes) 000367[1] 1786338294.906937 trunc from 0x7bfd37 to 0x4c09a7 (0x2ff38f bytes) 000368[0] 1786338294.962265 mapread 0x1e9f80 thru 0x1ea8d3 (0x954 bytes) 000370[0] 1786338295.060880 read 0x00b4ca thru 0x00ca31 (0x1568 bytes) 000372[0] 1786338295.095906 mapread 0x368143 thru 0x369b96 (0x1a54 bytes) 000375[0] 1786338295.145300 read 0x34a217 thru 0x34b607 (0x13f1 bytes) 000376[0] 1786338295.174605 read 0x3e6130 thru 0x3e6ec8 (0xd99 bytes) 000377[0] 1786338295.199791 trunc from 0x4c09a8 to 0x882c6a (0x3c22c3 bytes) 000378[1] 1786338295.302345 mapread 0x36981d thru 0x36a65a (0xe3e bytes) 000380[0] 1786338295.397776 read 0x5ef204 thru 0x5f0c52 (0x1a4f bytes) 000384[1] 1786338295.447837 trunc from 0x882c6b to 0x2e5787 (0x59d4e3 bytes) 000385[0] 1786338295.511883 mapwrite 0x64ee83 thru 0x650ba5 (0x1d23 bytes) 000387[0] 1786338295.720028 mapread 0x1352f5 thru 0x136df9 (0x1b05 bytes) 000388[1] 1786338295.747991 read 0x4b583c thru 0x4b5ba3 (0x368 bytes) 000390[0] 1786338295.820987 mapread 0x394b53 thru 0x395b98 (0x1046 bytes) 000391[0] 1786338295.887513 trunc from 0x650ba6 to 0x91062e (0x2bfa89 bytes) 000393[1] 1786338295.964614 trunc from 0x91062f to 0x433530 (0x4dd0fe bytes) 000396[1] 1786338296.054272 mapwrite 0x551d11 thru 0x551fbd (0x2ad bytes) 000397[0] 1786338296.144680 mapwrite 0x257c30 thru 0x259562 (0x1933 bytes) 000400[0] 1786338296.354578 read 0x055b99 thru 0x055c36 (0x9e bytes) 000402[1] 1786338296.418736 mapread 0x39caf8 thru 0x39d390 (0x899 bytes) 000404[1] 1786338296.459307 read 0x0472ca thru 0x047d8c (0xac3 bytes) 000407[1] 1786338296.507822 read 0x3a60ee thru 0x3a7fe1 (0x1ef4 bytes) 000409[0] 1786338296.564106 read 0x3b28e8 thru 0x3b2b57 (0x270 bytes) 000413[1] 1786338296.578772 write 0x047fb4 thru 0x04868b (0x6d8 bytes) 000415[1] 1786338296.630521 read 0x0142dd thru 0x0156ae (0x13d2 bytes) 000418[1] 1786338296.692763 read 0x1c69ce thru 0x1c88f0 (0x1f23 bytes) 000423[0] 1786338296.794501 write 0x335d05 thru 0x337d04 (0x2000 bytes) 000426[0] 1786338296.887403 read 0x10a545 thru 0x10bebb (0x1977 bytes) 000428[0] 1786338296.942369 read 0x016ba0 thru 0x016bff (0x60 bytes) 000430[0] 1786338297.091079 read 0x48d1f1 thru 0x48e022 (0xe32 bytes) 000431[1] 1786338297.132789 write 0x89b6b8 thru 0x89ccf5 (0x163e bytes) 000433[1] 1786338297.183699 write 0x31201f thru 0x312a00 (0x9e2 bytes) 000434[0] 1786338297.191856 mapread 0x2ba40c thru 0x2bb350 (0xf45 bytes) 000435[0] 1786338297.237640 write 0x53c653 thru 0x53cf28 (0x8d6 bytes) 000437[1] 1786338297.277519 mapwrite 0x6640a9 thru 0x6641c1 (0x119 bytes) 000438[0] 1786338297.362136 write 0x52b06f thru 0x52c46a (0x13fc bytes) 000442[1] 1786338297.483862 mapwrite 0x38f9f8 thru 0x391163 (0x176c bytes) 000443[1] 1786338297.642117 write 0x797a4a thru 0x7984ee (0xaa5 bytes) 000444[0] 1786338297.679807 read 0x166b7f thru 0x167cea (0x116c bytes) 000445[1] 1786338297.704743 mapread 0x6ca749 thru 0x6cb30f (0xbc7 bytes) 000448[0] 1786338297.766829 mapwrite 0x926eae thru 0x9280a7 (0x11fa bytes) 000450[1] 1786338297.994500 trunc from 0x9280a8 to 0x13c77d (0x7eb92a bytes) 000451[1] 1786338298.087732 mapwrite 0x03d8d4 thru 0x03dfad (0x6da bytes) 000459[0] 1786338298.332071 read 0x0122b7 thru 0x0133e7 (0x1131 bytes) 000460[0] 1786338298.432222 mapwrite 0x104212 thru 0x104776 (0x565 bytes) 000461[1] 1786338298.616580 write 0x9094eb thru 0x90a74c (0x1262 bytes) 000462[0] 1786338298.684030 write 0x4c4dd7 thru 0x4c523f (0x469 bytes) 000463[1] 1786338298.739909 mapwrite 0x6a8681 thru 0x6aa0a8 (0x1a28 bytes) 000468[1] 1786338298.959053 read 0x862415 thru 0x864180 (0x1d6c bytes) 000469[1] 1786338298.965803 mapwrite 0x38d102 thru 0x38e12d (0x102c bytes) 000472[0] 1786338299.110455 mapwrite 0x933547 thru 0x9343f9 (0xeb3 bytes) 000473[1] 1786338299.494334 mapread 0x46c40a thru 0x46d412 (0x1009 bytes) 000474[1] 1786338299.542589 mapread 0x1f4780 thru 0x1f59d6 (0x1257 bytes) 000476[1] 1786338299.597857 mapwrite 0x892f9a thru 0x893047 (0xae bytes) 000478[0] 1786338299.740698 write 0x071f3a thru 0x073897 (0x195e bytes) 000480[0] 1786338299.786004 mapwrite 0x7b8aa9 thru 0x7ba76b (0x1cc3 bytes) 000482[0] 1786338299.956079 read 0x8ca257 thru 0x8cb16c (0xf16 bytes) 000485[1] 1786338300.023905 mapwrite 0x673470 thru 0x6735b7 (0x148 bytes) 000486[0] 1786338300.123973 mapread 0x5929e9 thru 0x593c11 (0x1229 bytes) 000490[1] 1786338300.222716 trunc from 0x9343fa to 0x76c81b (0x1c7bde bytes) 000493[0] 1786338300.309652 trunc from 0x76c81c to 0x027c15 (0x744c06 bytes) 000495[1] 1786338300.464564 mapwrite 0x4bca6b thru 0x4be4bd (0x1a53 bytes) 000497[0] 1786338300.734855 mapread 0x009e9c thru 0x00a4ab (0x610 bytes) 000498[0] 1786338300.808101 write 0x3bf90d thru 0x3c0b62 (0x1256 bytes) 000499[0] 1786338300.864830 mapwrite 0x99e8bd thru 0x99e986 (0xca bytes) All operations completed A-OK! PASS 16b (33s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:05:06 (1786338306) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:05:11 (1786338311) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.679494 s, 12.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.416871 s, 20.1 MB/s ldlm.namespaces.MGC192.168.204.137@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f34c3021800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f34c558a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f34c3021800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f34c3021800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3002 s, 9.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.76399 s, 22.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.45995 s, 23.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.52702 s, 12.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.7028 s, 12.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.40174 s, 23.8 MB/s PASS 16d (61s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:06:12 (1786338372) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.356451 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105068 s, 10.0 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:06:21 (1786338381) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:06:51 (1786338411) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 01:07:22 (1786338442) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (10s) == sanityn test 16i: read after truncate file ============ 01:07:31 (1786338451) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 01:07:40 (1786338460) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.28714 s, 11.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96248 s, 13.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49837 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.35769 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40466 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.44653 s, 7.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.04192 s, 6.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64366 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.33154 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.18131 s, 5.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.39105 s, 4.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.24176 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.41914 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.74788 s, 7.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.93016 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.52817 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.61297 s, 5.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.88569 s, 5.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.00992 s, 5.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (53s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:08:33 (1786338513) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Aug 10 01:09:45 EDT 2026 + fsx 2 done: rc=0 Mon Aug 10 01:09:45 EDT 2026 + fsx 3 done: rc=0 Mon Aug 10 01:09:45 EDT 2026 + fsx 4 done: rc=0 Mon Aug 10 01:09:45 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Mon Aug 10 01:09:45 EDT 2026 PASS 16k (85s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:09:59 (1786338599) fail_loc=0x8000030a PASS 17 (15s) == sanityn test 18: mmap sanity check =========================================================================================== 01:10:14 (1786338614) mmap test1: basic mmap operation (PASS, 0.18897s) mmap test2: MAP_PRIVATE not write back (PASS, 0.13202s) mmap test3: concurrent mmap ops on two nodes (PASS, 34.573s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.633s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.18628s) mmap test6: check mmap write/read content on two nodes (PASS, 0.28813s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.6815s) mmap test11: multi-thread shared mmap (PASS, 1.2953s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1115s) PASS 18 (55s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:11:09 (1786338669) oleg437-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:11:15 (1786338675) PASS 20 (10s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:11:25 (1786338685) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (10s) == sanityn test 23: others should see updated atime while another read============================================================== 01:11:35 (1786338695) now is 1786338699 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 new atime is 1786338760 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:12:48 (1786338768) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (12s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:13:00 (1786338780) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:13:10 (1786338790) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (11s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:13:21 (1786338801) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (5s) == sanityn test 26a: allow mtime to get older ============ 01:13:26 (1786338806) utime: good mknod times 1786338807 <= 1786338808 <= 1786338808 for /mnt/lustre/f26a utime: good mknod times 1786338807 <= 1786338808 <= 1786338808 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (10s) == sanityn test 26b: sync mtime between ost and mds ====== 01:13:36 (1786338816) PASS 26b (14s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:13:51 (1786338831) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-10 01:13:53.000000000 -0400 Birth: 2026-08-10 01:13:53.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-10 01:13:53.000000000 -0400 Birth: 2026-08-10 01:13:53.000000000 -0400 PASS 26c (12s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:14:05 (1786338845) 0 PASS 30 (12s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:14:17 (1786338857) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:14:27 (1786338867) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.137@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (26s) == sanityn test 31r: open-rename(replace) race =========== 01:14:53 (1786338893) -rw-r--r-- 1 root root 0 Aug 10 01:14 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (14s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:15:07 (1786338907) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43190 Terminated tail -f $tfile PASS 31s (11s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:15:18 (1786338918) PASS 31t (13s) == sanityn test 32b: lockless i/o ======================== 01:15:31 (1786338931) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (6s) == sanityn test 32c: contention detection ================ 01:15:37 (1786338937) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.65609 s, 19.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81471 s, 28.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.43559 s, 15.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.92836 s, 13.3 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.75125 s, 14.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.92957 s, 13.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.28018 s, 23.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.34842 s, 22.3 MB/s PASS 32c (42s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:16:22 (1786338982) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:16:27 (1786338987) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:16:32 (1786338992) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:16:36 (1786338996) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (5s) == sanityn test 34: no lock timeout under IO ============= 01:16:40 (1786339000) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid in FULL state after 0 sec oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid in IDLE state after 0 sec oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid in FULL state after 0 sec oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid in IDLE state after 0 sec oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f34c3021800.ost_server_uuid in FULL state after 0 sec oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f34c3021800.ost_server_uuid in IDLE state after 0 sec PASS 34 (162s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:19:22 (1786339162) fail_loc=0x80000317 Race attempt 0 Wait for 50502 50521 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50502 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 571 (time 1786339183.63 total 10.02 last 57.01) - open/close 1094 (time 1786339193.63 total 20.02 last 52.26) - open/close 1686 (time 1786339203.64 total 30.03 last 59.18) - open/close 2288 (time 1786339213.64 total 40.03 last 60.19) - open/close 2881 (time 1786339223.66 total 50.05 last 59.17) - open/close 3400 (time 1786339233.67 total 60.06 last 51.85) - open/close 3708 (time 1786339243.67 total 70.06 last 30.79) total: 4000 open/close in 75.20 seconds: 53.19 ops/second PASS 35 (86s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:20:49 (1786339249) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg437-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58338 s, 33.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49234 s, 35.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0973 s, 47.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7432192) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47447 s, 35.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56831 s, 33.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.951986 s, 55.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7427072) after(7478272) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26536 s, 41.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7478272) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27532 s, 41.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7432192) after(7483392) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08248 s, 48.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7483392) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2258 s, 42.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34485 s, 39.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7439360) after(7497728) PASS 36 (341s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:26:30 (1786339590) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 - create 1164 (time 1786339605.32 total 10.00 last 116.38) - create 2258 (time 1786339615.33 total 20.01 last 109.33) - create 3360 (time 1786339625.33 total 30.01 last 110.18) - create 4508 (time 1786339635.33 total 40.01 last 114.75) - create 5602 (time 1786339645.34 total 50.02 last 109.30) - create 6718 (time 1786339655.34 total 60.03 last 111.58) - create 7814 (time 1786339665.36 total 70.04 last 109.47) - create 8873 (time 1786339675.37 total 80.05 last 105.75) - create 9849 (time 1786339685.37 total 90.05 last 97.57) total: 10000 create in 91.34 seconds: 109.48 ops/second PASS 37 (107s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:28:16 (1786339696) repeat after cancel_lru_locks PASS 39a (12s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:28:28 (1786339708) repeat after cancel_lru_locks PASS 39b (11s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:28:39 (1786339719) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 01:28:50 (1786339730) fail_loc=0x411 fail_loc=0 PASS 39d (10s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:29:02 (1786339742) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (23s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:29:25 (1786339765) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (23s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:29:48 (1786339788) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (23s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:30:11 (1786339811) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (23s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:30:34 (1786339834) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (18s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:30:52 (1786339852) open: File exists Conflict PASS 41b (21s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:31:14 (1786339874) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (20s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:31:33 (1786339893) Conflict PASS 41d (19s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:31:52 (1786339912) Conflict PASS 41e (19s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:32:11 (1786339931) Conflict PASS 41f (17s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:32:28 (1786339948) Conflict PASS 41g (20s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:32:48 (1786339968) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (18s) == sanityn test 41i: reint_open: create vs create ======== 01:33:07 (1786339987) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1210s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 01:53:18 (1786341198) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (17s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 01:53:34 (1786341214) open: File exists Conflict PASS 42b (18s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 01:53:52 (1786341232) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (20s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 01:54:12 (1786341252) Conflict PASS 42d (17s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 01:54:29 (1786341269) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (17s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 01:54:46 (1786341286) Conflict PASS 42f (18s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 01:55:04 (1786341304) Conflict PASS 42g (17s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 01:55:21 (1786341321) Conflict PASS 42h (18s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 01:55:39 (1786341339) PASS 43a (74s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 01:56:54 (1786341414) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 01:57:09 (1786341429) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 01:57:23 (1786341443) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 01:57:37 (1786341457) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 01:57:53 (1786341473) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 01:58:07 (1786341487) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 01:58:20 (1786341500) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 01:58:35 (1786341515) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 01:58:38 (1786341518) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (106s) == sanityn test 43k: unlink vs create ==================== 02:00:25 (1786341625) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (761s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:13:05 (1786342385) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Aug 10 02:13:10 EDT 2026 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:13:12 (1786342392) open: File exists Conflict PASS 44b (7s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:13:19 (1786342399) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (8s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:13:28 (1786342408) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:13:35 (1786342415) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:13:42 (1786342422) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:13:49 (1786342429) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:13:56 (1786342436) Conflict PASS 44h (7s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:14:03 (1786342443) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:14:04 (1786342444) PASS 45a (57s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:15:01 (1786342501) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:15:08 (1786342508) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:15:14 (1786342514) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:15:21 (1786342521) Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:15:29 (1786342529) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (8s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:15:37 (1786342537) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:15:44 (1786342544) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:15:51 (1786342551) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 02:15:52 (1786342552) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (530s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 02:24:42 (1786343082) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 02:24:48 (1786343088) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 02:24:53 (1786343093) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 02:24:59 (1786343099) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 02:25:05 (1786343105) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 02:25:11 (1786343111) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 02:25:17 (1786343117) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 02:25:22 (1786343122) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 02:25:28 (1786343128) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 02:25:29 (1786343129) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 02:25:30 (1786343130) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 02:25:31 (1786343131) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 02:25:33 (1786343133) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 02:25:34 (1786343134) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 02:25:35 (1786343135) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 02:25:36 (1786343136) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 02:25:37 (1786343137) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0101287 s, 1.0 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 02:25:45 (1786343145) 0+1 records in 0+1 records out 158 bytes copied, 0.0046839 s, 33.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218245) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 02:25:49 (1786343149) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00384185 s, 267 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144702 s, 72.5 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 02:26:05 (1786343165) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00349621 s, 293 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 02:26:14 (1786343174) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011971 s, 87.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 02:26:17 (1786343177) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 02:26:21 (1786343181) llite.lustre-ffff9f34c3021800.xattr_cache=1 llite.lustre-ffff9f34c558a800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:22.000000000 -0400 Modify: 2026-08-10 02:26:22.000000000 -0400 Change: 2026-08-10 02:26:22.000000000 -0400 Birth: 2026-08-10 02:26:22.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:27.000000000 -0400 Modify: 2026-08-10 02:26:28.000000000 -0400 Change: 2026-08-10 02:26:28.000000000 -0400 Birth: 2026-08-10 02:26:28.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:33.000000000 -0400 Modify: 2026-08-10 02:26:33.000000000 -0400 Change: 2026-08-10 02:26:33.000000000 -0400 Birth: 2026-08-10 02:26:33.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:33.000000000 -0400 Modify: 2026-08-10 02:26:33.000000000 -0400 Change: 2026-08-10 02:26:33.000000000 -0400 Birth: 2026-08-10 02:26:33.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:38.000000000 -0400 Modify: 2026-08-10 02:26:39.000000000 -0400 Change: 2026-08-10 02:26:39.000000000 -0400 Birth: 2026-08-10 02:26:39.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:26:38.000000000 -0400 Modify: 2026-08-10 02:26:39.000000000 -0400 Change: 2026-08-10 02:26:39.000000000 -0400 Birth: 2026-08-10 02:26:39.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 02:26:46 (1786343206) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 02:26:53 (1786343213) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 02:27:01 (1786343221) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 02:27:14 (1786343234) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 02:27:23 (1786343243) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 02:27:24 (1786343244) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 02:27:37 (1786343257) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 02:27:51 (1786343271) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1786343272 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.7745 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (25s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 02:28:16 (1786343296) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1786343298 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.69922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (38s) == sanityn test 60: Verify data_version behaviour ======== 02:28:54 (1786343334) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 02:28:57 (1786343337) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 02:28:59 (1786343339) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 02:29:02 (1786343342) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00212519 s, 30.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 02:29:04 (1786343344) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00414123 s, 15.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 02:29:05 (1786343345) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 02:29:06 (1786343346) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00439769 s, 14.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 02:29:07 (1786343347) llite.lustre-ffff9f34c3021800.xattr_cache=1 llite.lustre-ffff9f34c558a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 02:29:09 (1786343349) llite.lustre-ffff9f34c3021800.xattr_cache=1 llite.lustre-ffff9f34c558a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 02:29:12 (1786343352) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00656576 s, 156 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00295698 s, 346 kB/s 231001: get lock2 231005: get lock1 231005: done 231005: get lock2 231001: done 231001: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231005: cannot get lock2: Resource deadlock avoided 231005: exit rc=0 231001: done 231001: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 02:29:18 (1786343358) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00986483 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756226 s, 139 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231586 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 02:29:25 (1786343365) opening files: ........ closing files: ........ PASS 76 (15s) == sanityn test 77a: check FIFO NRS policy =============== 02:29:40 (1786343380) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0636294 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012276 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139581 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032871 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428695 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029456 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332655 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296996 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361464 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617594 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595161 s, 17.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545404 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601926 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507533 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487291 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553985 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105111 s, 99.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141427 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100353 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153524 s, 68.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150054 s, 69.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132739 s, 79.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106245 s, 98.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164363 s, 63.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143459 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00519287 s, 202 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246063 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258079 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258082 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130563 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360217 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856991 s, 122 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167828 s, 62.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 02:29:43 (1786343383) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0648394 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177087 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310476 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335976 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170907 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414132 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348505 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455552 s, 23.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457988 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589051 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437454 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289261 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032924 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458193 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627349 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603348 s, 17.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104938 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156229 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018807 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014268 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130903 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206534 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222671 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016217 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965246 s, 109 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138865 s, 75.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168197 s, 62.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435782 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643326 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130667 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036025 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498897 s, 210 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0657936 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114749 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223147 s, 47.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293133 s, 35.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278368 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218275 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191254 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251345 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367497 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337258 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584073 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481524 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406132 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559999 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593629 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396771 s, 26.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594438 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581914 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386866 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335422 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288458 s, 364 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330137 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358976 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639735 s, 164 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324901 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374923 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276312 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211844 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251185 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00653393 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329882 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355679 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483223 s, 217 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 02:29:46 (1786343386) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0951303 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123525 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183768 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201577 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140817 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405559 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325732 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411842 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346083 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433154 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360236 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046464 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495592 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399814 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264267 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467016 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605995 s, 17.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156471 s, 67.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288238 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221496 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252975 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01584 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221741 s, 47.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203785 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897469 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255284 s, 41.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320002 s, 32.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018929 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00529673 s, 198 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028808 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263001 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541112 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279198 s, 376 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.056372 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886022 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124856 s, 84.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230103 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305134 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271814 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240018 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430663 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273217 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020043 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540444 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459272 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268969 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515227 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228732 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366323 s, 28.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342177 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144292 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118325 s, 88.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822647 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179982 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174503 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232769 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607429 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209683 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234271 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377712 s, 278 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128193 s, 81.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106055 s, 98.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172683 s, 60.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028168 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114413 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537615 s, 195 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0545139 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00831449 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294586 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229029 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347556 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279603 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262581 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275055 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024992 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451125 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509726 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652078 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512835 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657017 s, 16.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515629 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669326 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793804 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126284 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154441 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108897 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194788 s, 53.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190016 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420854 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948516 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039711 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153562 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982565 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369521 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149712 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277136 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242712 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138637 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104397 s, 100 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 02:29:51 (1786343391) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132927 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106428 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276792 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316973 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391265 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294328 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459203 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435743 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479002 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303855 s, 34.5 MB/sldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443001 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300822 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380321 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379199 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387503 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658646 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426136 s, 24.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140292 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186313 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150665 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247141 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024088 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918569 s, 114 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722403 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014016 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285138 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173629 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212387 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276853 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567181 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181094 s, 57.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109068 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013692 s, 76.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.06345 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244096 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252637 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168119 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362129 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430921 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562943 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442006 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605901 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451359 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314112 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053007 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622541 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641848 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641373 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474612 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465568 s, 22.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199726 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214182 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191504 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186231 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192734 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231213 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015137 s, 69.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100044 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138928 s, 75.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836562 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010908 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292185 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166062 s, 63.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339356 s, 30.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198516 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130151 s, 80.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0607257 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155243 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146685 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336328 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248488 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379672 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366689 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430316 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179689 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152795 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394594 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427218 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590556 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413937 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343074 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523141 s, 20.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062502 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228722 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136792 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323148 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021201 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225188 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285463 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173383 s, 60.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188714 s, 55.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405645 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104598 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199126 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232242 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275494 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150569 s, 69.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110846 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153277 s, 68.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 02:29:55 (1786343395) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.37@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118161 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148546 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252532 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260659 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244245 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341588 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229451 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235581 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024169 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355401 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360512 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369209 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319424 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370734 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405272 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027523 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245707 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015383 s, 68.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183352 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195098 s, 53.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210402 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174773 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179439 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206758 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237012 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493438 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236304 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112119 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126855 s, 82.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196041 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110126 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188145 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297758 s, 35.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0498804 s, 336 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126618 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227821 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237341 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248241 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270157 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124878 s, 84.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236664 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289321 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146736 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219868 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363974 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033438 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393548 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429717 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354258 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374521 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143853 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114238 s, 91.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184374 s, 56.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013899 s, 75.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537322 s, 195 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109843 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871503 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050293 s, 208 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125417 s, 83.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624167 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150216 s, 69.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112099 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520344 s, 202 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753713 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870836 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446679 s, 235 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.059798 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00959221 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240647 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287035 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295225 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239901 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388967 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346812 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133979 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277709 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387637 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304855 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508199 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064048 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531715 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544439 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522858 s, 20.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125336 s, 83.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119259 s, 87.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153178 s, 68.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146192 s, 71.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01763 s, 59.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538123 s, 195 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173422 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149594 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202466 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032017 s, 328 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238624 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192221 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026194 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012853 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294491 s, 356 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385172 s, 272 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.111217 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153269 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198487 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265996 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252103 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336232 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224219 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630785 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424205 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656657 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049071 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448616 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716616 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607777 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439925 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594733 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436054 s, 24.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312981 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128801 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172301 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861135 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155762 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222793 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319482 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221231 s, 47.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214696 s, 48.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251283 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027545 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317493 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237756 s, 44.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307509 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200096 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269511 s, 38.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 02:30:03 (1786343403) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0788526 s, 213 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104785 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248845 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025948 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320114 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327441 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289084 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401523 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028453 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036453 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258412 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210204 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462831 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356354 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029461 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026692 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479173 s, 21.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100804 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187688 s, 55.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149862 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125883 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154282 s, 68.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126587 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162726 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190856 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146019 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219215 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184072 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218241 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221171 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240017 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025474 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238536 s, 44.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0874643 s, 192 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109067 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282184 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02993 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335009 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380609 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423688 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028541 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232959 s, 45.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03524 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351844 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409491 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541171 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439877 s, 23.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234611 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209337 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242419 s, 43.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114233 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149391 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209628 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212322 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146598 s, 71.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173715 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130545 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205962 s, 50.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197911 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177409 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213892 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017987 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388338 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053843 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520952 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00562044 s, 187 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.065531 s, 256 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101333 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291573 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264618 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182278 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156382 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390319 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215746 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450247 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026797 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236525 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492933 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521921 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503138 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558223 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507197 s, 20.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296605 s, 35.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172504 s, 60.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139194 s, 75.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016576 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176342 s, 59.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116279 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208573 s, 50.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245725 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614871 s, 171 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222351 s, 47.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690028 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123301 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520154 s, 202 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026216 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106396 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247477 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00589702 s, 178 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0693327 s, 242 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016896 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255209 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348962 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112302 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469744 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378377 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439481 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584078 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545355 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501874 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484126 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478801 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407825 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388027 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562362 s, 18.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c558a800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694231 s, 15.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134121 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118967 s, 88.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193857 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178447 s, 58.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209496 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355712 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157414 s, 66.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223995 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203768 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158661 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915285 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209044 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247775 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252136 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378707 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00503305 s, 208 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c558a800.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 02:30:10 (1786343410) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 02:30:13 (1786343413) oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 02:30:17 (1786343417) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 02:30:25 (1786343425) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0561782 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105415 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258845 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262622 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231525 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207358 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351956 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411391 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271328 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222576 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421786 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251345 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356045 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305572 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213944 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245887 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176413 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189214 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208471 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170352 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189796 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252615 s, 41.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195285 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218028 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195761 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205619 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242446 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210996 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233488 s, 44.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023383 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229887 s, 45.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245529 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250338 s, 41.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93997 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4345 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 02:31:11 (1786343471) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21922 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271301 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289636 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165405 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03024 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221634 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158967 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363143 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156721 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156146 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156922 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154733 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15092 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146475 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145815 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186256 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189034 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123375 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203619 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212214 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211759 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196954 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171273 s, 61.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197466 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164302 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041324 s, 254 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395894 s, 265 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236355 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237798 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857208 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269192 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232619 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554281 s, 189 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8301 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9648 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21489 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118736 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252223 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270852 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158704 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349798 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220367 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272111 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221265 s, 47.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169433 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169412 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166598 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170827 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168521 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185163 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202263 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18241 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165642 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190496 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179977 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021625 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224246 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188758 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226871 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190473 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244636 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00662046 s, 158 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211189 s, 49.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243609 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199097 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236764 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263234 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00233043 s, 450 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.835 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9604 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (136s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 02:33:27 (1786343607) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2156 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235923 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360814 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253524 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201727 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212619 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036899 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310353 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236366 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167127 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167049 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162107 s, 6.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164787 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184511 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162358 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171295 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184966 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200662 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180869 s, 58.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235713 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236225 s, 44.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233309 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262648 s, 39.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257479 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252494 s, 41.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337878 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159886 s, 65.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903456 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231641 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488911 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015133 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164034 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971345 s, 108 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8289 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9633 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21904 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421177 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411203 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351978 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161831 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489323 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033169 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357598 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051292 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147648 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147608 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130865 s, 8.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151137 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137934 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133616 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1565 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175584 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170712 s, 61.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241206 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025401 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282098 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279309 s, 37.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245175 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020524 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331655 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00618101 s, 170 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128262 s, 81.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251583 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344816 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163977 s, 63.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689447 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406643 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012435 s, 84.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8283 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9571 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (134s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 02:35:41 (1786343741) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246936 s, 67.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195323 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187427 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250687 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268045 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166788 s, 62.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281722 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259487 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354644 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503862 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539649 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554035 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439502 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429777 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362695 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390439 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289143 s, 36.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104095 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806217 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115147 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172215 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158775 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191317 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036197 s, 290 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045644 s, 230 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233197 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443312 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196029 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178223 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155275 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366976 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00341211 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531342 s, 197 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93329 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73198 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.37@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.37@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224165 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201187 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349484 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356408 s, 29.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335926 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041785 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404625 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313371 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286938 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184474 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372112 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925914 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358267 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245601 s, 4.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247093 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2959 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271078 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210756 s, 49.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177509 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017816 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174723 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022711 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184353 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024826 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232983 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227732 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952311 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026333 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502864 s, 209 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266645 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670647 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464175 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277209 s, 37.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93264 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72853 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.37@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224661 s, 74.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176064 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296298 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445183 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347665 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405755 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320326 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304698 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297951 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283901 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555744 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421629 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940321 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711563 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063712 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478388 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496551 s, 21.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196504 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198 s, 53.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020605 s, 50.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277738 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209625 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027069 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953049 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219764 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289914 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289414 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327971 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688063 s, 152 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160544 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153063 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161734 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086198 s, 122 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94212 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87893 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424922 s, 39.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176778 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273798 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195903 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431659 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409983 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420112 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04199 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032092 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618935 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794462 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662633 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461604 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294584 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291641 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362756 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337555 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158221 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153211 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011528 s, 91.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200063 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107542 s, 97.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539302 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172149 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156085 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209006 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201885 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456541 s, 230 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439677 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450095 s, 233 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024868 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425103 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494789 s, 212 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78472 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73081 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90693 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.820449 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305157 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285237 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414131 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396855 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501251 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433398 s, 24.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484348 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648591 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166557 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185125 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161985 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171921 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185859 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193029 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173829 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190926 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066678 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159683 s, 65.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119918 s, 87.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162242 s, 64.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154387 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177294 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018711 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200517 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565142 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115995 s, 90.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593625 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132629 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387216 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00830158 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003234 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302117 s, 347 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6712 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4309 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227132 s, 73.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130941 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267946 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342457 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555346 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569592 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594897 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486431 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015245 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321703 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341435 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572984 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259337 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580176 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644239 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044606 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504375 s, 20.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210125 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02062 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168304 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110313 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164681 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163002 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664882 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211452 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023924 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239472 s, 43.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217494 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237795 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257705 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021468 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00334174 s, 314 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415389 s, 252 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93884 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73022 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42631 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011161 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263127 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193711 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325279 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272966 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306057 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200797 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225822 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562522 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647196 s, 16.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526506 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893338 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378247 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372644 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413869 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415057 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181578 s, 57.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183721 s, 57.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204261 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219946 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162985 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016803 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160985 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221755 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274101 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254526 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987521 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024273 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255036 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422282 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00696491 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238052 s, 440 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79672 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73002 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76158 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73238 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (356s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 02:41:37 (1786344097) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.235812 s, 71.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128448 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271527 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292107 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267185 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333356 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248948 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288142 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342057 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256938 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273003 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035671 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041916 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058426 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491886 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324583 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528571 s, 19.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340168 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131229 s, 79.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149792 s, 70.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110312 s, 95.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166087 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138417 s, 75.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138502 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035547 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171635 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016131 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187926 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232117 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210872 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357776 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415922 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038923 s, 269 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93402 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73391 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 02:42:13 (1786344133) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.37@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.37@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222445 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172781 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128859 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330123 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381539 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351984 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351393 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214392 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152241 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033369 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411265 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432642 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391297 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303093 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416951 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253989 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258756 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137143 s, 76.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159927 s, 65.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159604 s, 65.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133235 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240147 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252974 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112014 s, 93.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207722 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181264 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355128 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111507 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381447 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382836 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571966 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385048 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336935 s, 311 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94234 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73185 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (40s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 02:42:53 (1786344173) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.37@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.243152 s, 69.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148003 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262933 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027686 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259747 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396654 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272031 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327765 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465577 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196707 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399551 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048489 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263059 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504325 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465049 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252531 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238059 s, 44.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200583 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134421 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230991 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011544 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174049 s, 60.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267156 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235711 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218492 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029001 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170628 s, 61.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299052 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665537 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261958 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535513 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499085 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447719 s, 234 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93833 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96836 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 02:43:25 (1786344205) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424859 s, 39.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029621 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484229 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040213 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398073 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618962 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406977 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038598 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583656 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475706 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803625 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606186 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718817 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243524 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268675 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31229 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303376 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179039 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177732 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154162 s, 68.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193963 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137725 s, 76.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191805 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168792 s, 62.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145968 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424252 s, 247 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438415 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097606 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429367 s, 244 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486077 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389946 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00974984 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349512 s, 300 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79196 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73226 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92168 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72965 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 02:44:46 (1786344286) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.841175 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161187 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291596 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226341 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340307 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317022 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286286 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268949 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271974 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163865 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178479 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167256 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174572 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170711 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164667 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188132 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165538 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154559 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198069 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207804 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288266 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599395 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181381 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318471 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135244 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334467 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180104 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280088 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129086 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354172 s, 29.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157301 s, 66.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176032 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139278 s, 75.3 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6741 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9438 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 02:45:52 (1786344352) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254255 s, 66.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148184 s, 70.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030432 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263538 s, 39.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031975 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380444 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329017 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286842 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289387 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420702 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037655 s, 27.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386547 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365071 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273032 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420328 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457223 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401951 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135943 s, 77.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175141 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166391 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142461 s, 73.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162058 s, 64.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207743 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133846 s, 78.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212521 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871714 s, 120 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020486 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254643 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104271 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593231 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411579 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096042 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716175 s, 146 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9448 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73326 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg437-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430658 s, 39.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177895 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287063 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287843 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263799 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187288 s, 56.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221233 s, 47.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297573 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219719 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540527 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906483 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105141 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959348 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354286 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368272 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430614 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43246 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183413 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168418 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180531 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166934 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130164 s, 80.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189082 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173063 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223357 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266358 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116566 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047036 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782434 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834553 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799669 s, 131 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149228 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148958 s, 70.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8314 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73379 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76669 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73184 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (117s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 02:47:49 (1786344469) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:) Stopping client oleg437-client.virtnet /mnt/lustre opts: Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg437-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg437-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg437-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.37,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg437-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.37,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg437-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.37,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/67581/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.37,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg437-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Starting client oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Started clients oleg437-client.virtnet: 192.168.204.137@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Starting client oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 Started clients oleg437-client.virtnet: 192.168.204.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f34c304a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f34c6026800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c304a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c6026800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246198 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033138 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618997 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453094 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117915 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123357 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111414 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132211 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144514 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129136 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12845 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147372 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140048 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234689 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252659 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278975 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285052 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202595 s, 51.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258762 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251329 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285389 s, 36.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284813 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266014 s, 39.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033335 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031208 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308017 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318462 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301905 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300655 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316706 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350648 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305836 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303178 s, 34.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg437-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93787 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96735 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253837 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432435 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300872 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720717 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153647 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143182 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147599 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143814 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162102 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140116 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163855 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167344 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179722 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247935 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260687 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284647 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300733 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140937 s, 74.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212764 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196606 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168233 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195129 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199278 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201488 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251933 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200337 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147731 s, 71.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733231 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234854 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928677 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486663 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343717 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00734851 s, 143 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93441 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73334 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (114s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 02:49:43 (1786344583) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 02:49:51 (1786344591) On MGS 192.168.204.137, active = nodemap.active=1 On MGS 192.168.204.137, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254877 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361608 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288592 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847804 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082665 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871641 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113114 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104216 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103956 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12499 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111601 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100949 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117152 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26238 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260691 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289341 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308189 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176184 s, 59.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154427 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018962 s, 55.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224123 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018309 s, 57.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244923 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179089 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226816 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229378 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00840015 s, 125 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914429 s, 115 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278707 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316599 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272183 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284698 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226764 s, 462 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93579 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96807 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.137, TBF.id = On MGS 192.168.204.137, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154579 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411907 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375719 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144354 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158766 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147101 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162728 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15384 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1511 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17418 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175825 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162548 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196564 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213367 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188405 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201717 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298958 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331173 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220605 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296563 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336766 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408465 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034587 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354176 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503724 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361425 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319841 s, 32.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318776 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00873418 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386406 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146062 s, 71.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429844 s, 24.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c6026800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9f34c6026800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356335 s, 294 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.82119 s, 57.6 MB/s Write runtime is 3 s, speed is 33.333333 IOPS write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.512734 s, 205 MB/s Read runtime is 2 s, speed is 50.000000 IOPS read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg437-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 On MGS 192.168.204.137, active = nodemap.active=1 On MGS 192.168.204.137, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260136 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026787 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039172 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511173 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566231 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063207 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582166 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560189 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736094 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542562 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119101 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137426 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133666 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136337 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136437 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124646 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143348 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149266 s, 70.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165947 s, 63.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01953 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217866 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242973 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199059 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256022 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193463 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013036 s, 80.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00792232 s, 132 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247225 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733149 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115401 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158261 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294277 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00208881 s, 502 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94192 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73428 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.137, TBF.id = On MGS 192.168.204.137, active = nodemap.active=0 PASS 77kj (118s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 02:51:49 (1786344709) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 02:51:53 (1786344713) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg437-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47908 s, 301 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg437-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0041 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 02:52:43 (1786344763) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136717 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264713 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033238 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551494 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673397 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762013 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625595 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709183 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591311 s, 17.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f34c6026800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703499 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110743 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11535 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115741 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11799 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114619 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132089 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142024 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f34c6026800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481184 s, 218 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014889 s, 70.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171445 s, 61.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180909 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01706 s, 61.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176509 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234255 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137398 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195742 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406145 s, 258 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235101 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441407 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228484 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419184 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026942 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046007 s, 228 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94821 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96947 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244485 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285741 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416006 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055651 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569583 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597945 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742715 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750497 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809505 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916677 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113103 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102432 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901209 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100476 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918807 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249377 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240454 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208942 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219815 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021134 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211002 s, 49.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205802 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195902 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240021 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00775713 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178059 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021861 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021238 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00646716 s, 162 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299612 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243085 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281564 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00182858 s, 573 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg437-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg437-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94586 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97093 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 02:53:48 (1786344828) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 02:53:52 (1786344832) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (36s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 02:54:28 (1786344868) mds.MDS.mdt.nrs_policies=tbf oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg437-client: oleg437-server: ssh exited with exit code 36 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg437-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (11s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 02:54:39 (1786344879) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 276611 kill md_thread xargs PASS 77r (41s) == sanityn test 77s: Check TBF LRU shrinker ============== 02:55:20 (1786344920) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (12s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 02:55:32 (1786344932) oleg437-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg437-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg437-client: oleg437-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (2s) == sanityn test 79: xattr: intent error ================== 02:55:35 (1786344935) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936387 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:55:35.000000000 -0400 Modify: 2026-08-10 02:55:35.000000000 -0400 Change: 2026-08-10 02:55:35.000000000 -0400 Birth: 2026-08-10 02:55:36.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 02:55:47 (1786344947) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 02:55:48 (1786344948) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 02:55:49 (1786344949) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 02:55:50 (1786344950) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 02:55:52 (1786344952) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 02:55:53 (1786344953) total: 2500 open/close in 5.84 seconds: 427.96 ops/second snapshot_time 1786344977.169458620 secs.nsecs start_time 1786344954.576999450 secs.nsecs elapsed_time 22.592459170 secs.nsecs open 2505 samples [usecs] 100 2741 1067696 498521644 close 2504 samples [usecs] 68 2218 280510 42118334 mknod 2501 samples [usecs] 241 2658 952169 401222993 unlink 2500 samples [usecs] 327 11841 1515402 1602312258 mkdir 2 samples [usecs] 792 1046 1838 1721380 rmdir 2 samples [usecs] 2115 3494 5609 16681261 rename 2500 samples [usecs] 298 11140 3206571 5794793141 getattr 3760 samples [usecs] 7 3260 82938 13080970 parallel_rename_file 2500 samples [usecs] 298 11140 3206571 5794793141 crossdir_rename 2500 samples [usecs] 298 11140 3206571 5794793141 readpage 12 samples [usecs] 1978 14807 47858 327336712 PASS 81d (25s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 02:56:18 (1786344978) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7623 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7623 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 02:56:20 (1786344980) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 02:56:21 (1786344981) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 02:56:28 (1786344988) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1786344988 Finishing test test1 at 1786344989 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 02:56:30 (1786344990) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 02:56:31 (1786344991) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 02:56:32 (1786344992) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 02:56:33 (1786344993) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6538 0x198a 0x240000400 1 6500 0x1964 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6539 0x198b 0x240000400 1 6501 0x1965 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 02:56:42 (1786345002) osc.lustre-OST0000-osc-ffff9f34c304a000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9f34c6026800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f34c304a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f34c6026800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0350935 s, 233 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09726 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 298156 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130849 s, 313 kB/s osc.lustre-OST0000-osc-ffff9f34c304a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f34c6026800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c304a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f34c6026800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 02:56:52 (1786345012) llite.lustre-ffff9f34c304a000.fast_read=0 llite.lustre-ffff9f34c6026800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0164933 s, 127 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9f34c304a000.fast_read=1 llite.lustre-ffff9f34c6026800.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 02:57:07 (1786345027) llite.lustre-ffff9f34c304a000.fast_read=0 llite.lustre-ffff9f34c6026800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00176772 s, 7.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9f34c304a000.fast_read=1 llite.lustre-ffff9f34c6026800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 02:57:15 (1786345035) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 02:57:16 (1786345036) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00135922 s, 3.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 02:57:18 (1786345038) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0351439 s, 59.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 02:57:20 (1786345040) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0388721 s, 54.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 02:57:22 (1786345042) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00146545 s, 8.2 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 02:57:24 (1786345044) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00153642 s, 2.7 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 02:57:26 (1786345046) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00137816 s, 3.0 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 02:57:28 (1786345048) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00127435 s, 3.2 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 02:57:31 (1786345051) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 24 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x1224:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (2s) == sanityn test 103: Test size correctness with lockahead ========================================================== 02:57:33 (1786345053) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1786345054 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1786345056 Finishing test test23 at 1786345056 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 02:57:40 (1786345060) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 02:57:41 (1786345061) fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 02:57:58 (1786345078) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 02:58:00 (1786345080) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00382463 s, 268 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:58:00.000000000 -0400 Modify: 2026-08-10 02:58:00.000000000 -0400 Change: 2026-08-10 02:58:00.000000000 -0400 Birth: 2026-08-10 02:58:01.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938923 -rw-r--r-- 1 0 0 1786345081 1786345080 1786345080 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:58:00.000000000 -0400 Modify: 2026-08-10 02:58:00.000000000 -0400 Change: 2026-08-10 02:58:00.000000000 -0400 Birth: 2026-08-10 02:58:01.000000000 -0400 1786345080 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 02:58:02 (1786345082) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 02:58:04 (1786345084) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0877116 s, 120 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6548 0x1994 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 02:58:08 (1786345088) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0875795 s, 120 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6508 0x196c 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 02:58:17 (1786345097) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00208879 s, 4.8 MB/s llite.lustre-ffff9f34c304a000.hybrid_io=0 llite.lustre-ffff9f34c6026800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.20956 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 02:58:23 (1786345103) 192.168.204.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 192.168.204.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg437-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 192.168.204.137@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg437-client.virtnet /mnt/lustre (opts:) Stopping client oleg437-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre2 PASS 109 (36s) == sanityn test 110: do not grant another lock on resend ========================================================== 02:58:59 (1786345139) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 02:58:59.000000000 -0400 Modify: 2026-08-10 02:58:59.000000000 -0400 Change: 2026-08-10 02:58:59.000000000 -0400 Birth: 2026-08-10 02:59:00.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg437-client.virtnet oleg437-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 03:01:45 (1786345305) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 03:01:46 (1786345306) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 03:01:47 (1786345307) lustre-OST0000-osc-ffff9f34c8988000 active. lustre-OST0001-osc-ffff9f34c8988000 active. lustre-OST0000-osc-ffff9f34c8988000 active. lustre-OST0001-osc-ffff9f34c8988000 active. MGC192.168.204.137@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 03:01:49 (1786345309) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 03:01:50 (1786345310) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 03:01:51 (1786345311) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 03:01:52 (1786345312) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00279377 s, 7.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128924 s, 1.6 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 03:01:57 (1786345317) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0874 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0901 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1079 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0935 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.098 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1004 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0975 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1472 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1544 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1572 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1728 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1664 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1684 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1735 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2507 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.248 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2514 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2363 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f34c8988000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f34c8988000.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 7705 sec ============== 03:03:04 (1786345384) === sanityn: start cleanup 03:03:05 (1786345385) === Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg437-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 03:03:55 (1786345435) ===