-----============= acceptance-small: sanityn ============----- Sat Sep 5 10:46:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg455-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 10:46:37 (1788619597) === Starting client oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 Mount client oleg455-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 Started clients oleg455-client.virtnet: 192.168.204.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg455-client.virtnet: executing check_config_client /mnt/lustre oleg455-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg455-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9af283332800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af285452800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af283332800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af285452800.idle_timeout=debug disable quota as required oleg455-server: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 10:47:25 (1788619645) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303237 s, 34.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7654] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 10:47:27 (1788619647) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 10:47:35 (1788619655) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 10:47:42 (1788619662) -rw-r--r-- 1 root root 0 Sep 5 10:47 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 10:47:49 (1788619669) -rw-r--r-- 1 root root 0 Sep 5 10:47 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 10:47:55 (1788619675) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 10:48:03 (1788619683) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 10:48:11 (1788619691) total 1 -rwxrwxrwx 1 root root 0 Sep 5 10:47 f2a -rwxrwxrwx 1 root root 0 Sep 5 10:47 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 5 10:47 f2a -rwxrwxrwx 1 root root 0 Sep 5 10:47 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0121188 s, 42.2 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 10:48:18 (1788619698) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 10:48:22 (1788619702) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.523724 s, 4.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 10:48:29 (1788619709) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 10:48:37 (1788619717) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 10:48:46 (1788619726) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 10:48:54 (1788619734) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 10:49:01 (1788619741) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 10:49:09 (1788619749) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 10:49:16 (1788619756) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 10:49:25 (1788619765) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669213 s, 15.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0669508 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 10:49:32 (1788619772) 1+0 records in 1+0 records out 1 byte copied, 0.00725026 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.019431 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0239203 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0281159 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201636 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113656 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102355 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0099072 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0206153 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.020327 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0162131 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0197023 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 10:49:40 (1788619780) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.044893 s, 68.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0536061 s, 76.4 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00020077 s, 15.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 10:49:47 (1788619787) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 10:49:55 (1788619795) warning: '-runtime' deprecated, use '-t runtime' instead - link 313 (time 1788619807.74 total 10.00 last 31.29) - link 617 (time 1788619817.80 total 20.06 last 30.21) - link 906 (time 1788619827.82 total 30.08 last 28.84) - link 1228 (time 1788619837.88 total 40.14 last 32.03) - link 1522 (time 1788619847.89 total 50.15 last 29.37) - link 1822 (time 1788619857.90 total 60.16 last 29.95) - link 2114 (time 1788619867.92 total 70.18 last 29.17) - link 2386 (time 1788619877.94 total 80.20 last 27.13) - link 2674 (time 1788619887.97 total 90.23 last 28.72) total: 2957 link in 100.05 seconds: 29.55 ops/second using seed 2218300466 running for 100 seconds - stat 4584 (time 1788619809 ; total 11 ; last 11) - stat 9358 (time 1788619820 ; total 22 ; last 11) - stat 10000 (time 1788619821 ; total 23 ; last 1) - stat 15316 (time 1788619832 ; total 34 ; last 11) - stat 20000 (time 1788619841 ; total 43 ; last 9) - stat 24195 (time 1788619852 ; total 54 ; last 11) - stat 26672 (time 1788619863 ; total 65 ; last 11) - stat 29064 (time 1788619874 ; total 76 ; last 11) - stat 30000 (time 1788619878 ; total 80 ; last 4) - stat 32308 (time 1788619889 ; total 91 ; last 11) total: 34245 stats in 100 seconds: 342.450012 stats/second - unlinked 0 (time 1788619848 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2957) error: No such file or directory total: 2956 unlinks in 88 seconds: 33.590908 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21463) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21465) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 10:52:23 (1788619943) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 10:52:32 (1788619952) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22654 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 10:52:40 (1788619960) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23233 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 10:52:47 (1788619967) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23812 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 10:52:55 (1788619975) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24394 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 10:53:02 (1788619982) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24976 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 10:53:12 (1788619992) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (57s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 10:54:08 (1788620048) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.566875 s, 14.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.551165 s, 15.2 MB/s Chance of close/open is 1 in 50 Seed set to 54 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole truncating to largest ever: 0x8cafa3 000002[0] 1788620055.444516 trunc from 00000000 to 0x8cafa2 (0x8cafa3 bytes) 000003[0] 1788620055.516455 mapwrite 0x289b15 thru 0x28b3c1 (0x18ad bytes) 000005[0] 1788620055.674072 punch from 0x310f82 to 0x3167b0 (0x582f bytes) 000007[0] 1788620055.715488 punch from 0x264b6e to 0x26794d (0x2de0 bytes) 000008[1] 1788620055.721322 mapwrite 0x9619e4 thru 0x96f1c5 (0xd7e2 bytes) 000011[1] 1788620056.077117 mapwrite 0x3d8ce7 thru 0x3e86b6 (0xf9d0 bytes) 000012[1] 1788620056.225418 write 0x707b19 thru 0x70cfcb (0x54b3 bytes) 000013[0] 1788620056.278782 mapread 0x8d28be thru 0x8d31e9 (0x92c bytes) 000014[0] 1788620056.336112 write 0x5ad29f thru 0x5b8b1e (0xb880 bytes) 000016[0] 1788620056.413916 punch from 0x4c42b2 to 0x4ced2a (0xaa79 bytes) 000017[1] 1788620056.435110 mapwrite 0x876825 thru 0x879039 (0x2815 bytes) 000018[1] 1788620056.608777 punch from 0x06992b to 0x06d9c2 (0x4098 bytes) 000021[0] 1788620056.709011 mapread 0x29e7ce thru 0x2a8723 (0x9f56 bytes) 000022[1] 1788620056.768883 mapwrite 0x23a17e thru 0x2415a2 (0x7425 bytes) 000023[1] 1788620056.874547 mapwrite 0x9b766e thru 0x9ba923 (0x32b6 bytes) 000024[0] 1788620057.003234 mapwrite 0x1abac3 thru 0x1acb94 (0x10d2 bytes) 000025[1] 1788620057.113565 write 0x146f42 thru 0x1530f9 (0xc1b8 bytes) 000026[0] 1788620057.163084 read 0x46bf58 thru 0x476b9c (0xac45 bytes) 000028[1] 1788620057.202183 mapwrite 0x9e82c2 thru 0x9f063c (0x837b bytes) 000029[0] 1788620057.362770 write 0x799683 thru 0x7a0d3d (0x76bb bytes) 000031[0] 1788620057.431760 write 0x663b7c thru 0x66e51f (0xa9a4 bytes) 000032[0] 1788620057.470393 mapread 0x59e440 thru 0x5a2e86 (0x4a47 bytes) 000034[1] 1788620057.719844 trunc from 0x9f063d to 0x5f783e (0x3f8dfe bytes) 000035[1] 1788620057.854803 mapread 0x2b09b3 thru 0x2b792d (0x6f7b bytes) 000036[1] 1788620057.980796 mapwrite 0x74e71f thru 0x75adff (0xc6e1 bytes) 000037[1] 1788620058.238611 write 0x6b2b50 thru 0x6b6027 (0x34d8 bytes) 000039[1] 1788620058.261552 mapwrite 0x1accec thru 0x1b89fb (0xbd10 bytes) 000041[1] 1788620058.398322 read 0x0f243f thru 0x0f34b7 (0x1079 bytes) 000046[0] 1788620058.472201 write 0x9bd50f thru 0x9c72a4 (0x9d96 bytes) truncating to largest ever: 0x964462 000047[1] 1788620058.505221 trunc from 0x9c72a5 to 0x964461 (0x62e43 bytes) 000049[1] 1788620058.596153 mapwrite 0x0ce4dc thru 0x0dcc8e (0xe7b3 bytes) 000050[0] 1788620058.738466 write 0x9ff2db thru 0x9fffff (0xd25 bytes) 000051[0] 1788620058.773866 punch from 0x1ef373 to 0x1fdaa0 (0xe72e bytes) 000055[1] 1788620058.878201 mapread 0x2819bb thru 0x28e887 (0xcecd bytes) 000056[1] 1788620058.965373 mapread 0x968271 thru 0x96a0d9 (0x1e69 bytes) 000057[0] 1788620058.985783 mapread 0x13aceb thru 0x143459 (0x876f bytes) 000062[1] 1788620059.141202 punch from 0x17d77e to 0x189919 (0xc19c bytes) 000063[1] 1788620059.210368 read 0x2aaeea thru 0x2b8e5b (0xdf72 bytes) 000064[1] 1788620059.227928 write 0x29ffbe thru 0x2a6c07 (0x6c4a bytes) 000066[1] 1788620059.240333 read 0x2dd155 thru 0x2ec632 (0xf4de bytes) 000068[0] 1788620059.281519 write 0x2971e7 thru 0x2a5f0e (0xed28 bytes) 000069[0] 1788620059.392014 trunc from 0xa00000 to 0x32d2a9 (0x6d2d56 bytes) 000071[1] 1788620059.508210 mapread 0x201eeb thru 0x211105 (0xf21b bytes) 000074[1] 1788620059.675574 write 0x7bb330 thru 0x7c929d (0xdf6e bytes) 000075[1] 1788620059.726070 mapread 0x2dd2b8 thru 0x2e3332 (0x607b bytes) 000078[1] 1788620059.777717 mapwrite 0x8adb40 thru 0x8b991a (0xbddb bytes) 000080[0] 1788620059.998704 mapread 0x80eaae thru 0x816ad7 (0x802a bytes) 000082[0] 1788620060.091052 mapread 0x7fbc6d thru 0x8050e4 (0x9478 bytes) 000083[0] 1788620060.172852 write 0x57d490 thru 0x58653f (0x90b0 bytes) 000084[0] 1788620060.240755 mapwrite 0x20597a thru 0x212d5f (0xd3e6 bytes) 000089[1] 1788620060.480860 punch from 0x1dccf3 to 0x1ea5d9 (0xd8e7 bytes) 000091[0] 1788620060.549034 punch from 0x6c551b to 0x6d5487 (0xff6d bytes) 000092[0] 1788620060.571917 punch from 0x64bfa5 to 0x65b143 (0xf19f bytes) 000093[0] 1788620060.597027 write 0x530766 thru 0x539cd8 (0x9573 bytes) 000094[1] 1788620060.629359 mapwrite 0x69f969 thru 0x6a530f (0x59a7 bytes) 000097[0] 1788620060.842017 mapread 0x873174 thru 0x876ec1 (0x3d4e bytes) 000099[0] 1788620060.938371 read 0x74c55b thru 0x74d77c (0x1222 bytes) 000100[1] 1788620060.993181 read 0x3adf25 thru 0x3b4dd7 (0x6eb3 bytes) 000101[0] 1788620061.011751 mapwrite 0x1f0201 thru 0x1f7ac7 (0x78c7 bytes) 000102[1] 1788620061.112116 mapread 0x838af3 thru 0x83f757 (0x6c65 bytes) 000103[0] 1788620061.166432 write 0x6c4db6 thru 0x6cd41c (0x8667 bytes) 000104[0] 1788620061.201576 punch from 0x045d1b to 0x04ac71 (0x4f57 bytes) 000105[1] 1788620061.237432 mapwrite 0x6e382b thru 0x6f27ec (0xefc2 bytes) 000106[0] 1788620061.471275 read 0x2fe967 thru 0x30830b (0x99a5 bytes) 000107[0] 1788620061.519137 write 0x1a38ca thru 0x1a3c18 (0x34f bytes) 000109[0] 1788620061.560257 mapread 0x861e76 thru 0x865d57 (0x3ee2 bytes) 000110[1] 1788620061.613563 mapwrite 0x75da2e thru 0x761ff3 (0x45c6 bytes) 000111[1] 1788620061.740306 punch from 0x09a2ea to 0x0a4c84 (0xa99b bytes) 000112[0] 1788620061.772374 write 0x06b23f thru 0x0716cb (0x648d bytes) 000113[0] 1788620061.846214 mapread 0x093a1a thru 0x09b659 (0x7c40 bytes) 000117[1] 1788620062.050917 mapread 0x5306ea thru 0x5342ba (0x3bd1 bytes) 000118[0] 1788620062.137387 mapwrite 0x58d29b thru 0x597d0c (0xaa72 bytes) 000119[1] 1788620062.270367 read 0x48fca5 thru 0x491739 (0x1a95 bytes) 000120[0] 1788620062.287963 write 0x7de7e5 thru 0x7e6e86 (0x86a2 bytes) 000121[0] 1788620062.360468 mapread 0x00f503 thru 0x012fb1 (0x3aaf bytes) 000122[1] 1788620062.384488 trunc from 0x8b991b to 0x898a26 (0x20ef4 bytes) 000123[0] 1788620062.478916 trunc from 0x898a27 to 0x45b438 (0x43d5ee bytes) 000124[0] 1788620062.526155 mapwrite 0x914461 thru 0x921a98 (0xd638 bytes) 000125[1] 1788620062.712290 read 0x8b09b7 thru 0x8b7721 (0x6d6b bytes) 000126[1] 1788620062.766504 mapwrite 0x207eab thru 0x20c41f (0x4575 bytes) 000127[1] 1788620062.881032 read 0x5de7cf thru 0x5ec8d6 (0xe108 bytes) 000128[0] 1788620062.932846 read 0x60a15f thru 0x60a3cd (0x26f bytes) 000129[0] 1788620062.946019 mapread 0x2e14e9 thru 0x2e17a7 (0x2bf bytes) 000130[0] 1788620062.983487 mapwrite 0x9bb370 thru 0x9c1387 (0x6018 bytes) 000132[1] 1788620063.207805 read 0x53d67e thru 0x53e6fb (0x107e bytes) 000133[1] 1788620063.268753 punch from 0x195415 to 0x19b7c4 (0x63b0 bytes) 000134[1] 1788620063.339949 trunc from 0x9c1388 to 0x37e45e (0x642f29 bytes) 000135[1] 1788620063.449183 mapread 0x21aa26 thru 0x222e8d (0x8468 bytes) 000136[1] 1788620063.497623 mapwrite 0x690a39 thru 0x69a04e (0x9616 bytes) 000137[0] 1788620063.621348 read 0x31b20e thru 0x322b23 (0x7916 bytes) 000140[1] 1788620063.695073 mapread 0x12fb3b thru 0x13d26d (0xd733 bytes) 000141[0] 1788620063.756037 mapwrite 0x919cc3 thru 0x921723 (0x7a61 bytes) 000142[1] 1788620063.982468 write 0x9ee02e thru 0x9fa0bc (0xc08f bytes) 000145[1] 1788620064.057198 mapwrite 0x0b6e39 thru 0x0bf6e3 (0x88ab bytes) 000147[1] 1788620064.221931 punch from 0x75a4db to 0x75bd1b (0x1841 bytes) 000149[1] 1788620064.245570 trunc from 0x9fa0bd to 0x96219d (0x97f1f bytes) 000152[0] 1788620064.377200 trunc from 0x96219e to 0x21bffa (0x7461a3 bytes) 000153[1] 1788620064.477519 mapread 0x006206 thru 0x00ba95 (0x5890 bytes) 000155[1] 1788620064.609132 mapwrite 0x0ffb4f thru 0x10e002 (0xe4b4 bytes) 000156[1] 1788620064.723115 read 0x106d15 thru 0x110f74 (0xa260 bytes) 000158[0] 1788620064.754652 read 0x025864 thru 0x0264ac (0xc49 bytes) 000159[1] 1788620064.838712 punch from 0x1f9392 to 0x203fe1 (0xac50 bytes) 000162[1] 1788620064.975861 write 0x851a6d thru 0x851eec (0x480 bytes) 000163[1] 1788620065.030125 write 0x5f642a thru 0x5f78a6 (0x147d bytes) 000164[0] 1788620065.167320 mapread 0x0c6452 thru 0x0cef11 (0x8ac0 bytes) 000165[1] 1788620065.355497 mapread 0x18746a thru 0x193fc3 (0xcb5a bytes) 000167[0] 1788620065.539735 mapread 0x1f3494 thru 0x1f7fb4 (0x4b21 bytes) 000169[1] 1788620065.600089 read 0x4a299a thru 0x4ab01f (0x8686 bytes) 000171[1] 1788620065.603779 write 0x6f84a8 thru 0x6f970a (0x1263 bytes) 000172[0] 1788620065.605211 punch from 0x1a39ab to 0x1acbbb (0x9211 bytes) 000174[0] 1788620065.683293 write 0x71aea2 thru 0x71bcab (0xe0a bytes) 000175[1] 1788620065.791222 mapread 0x524e81 thru 0x52aac4 (0x5c44 bytes) 000176[1] 1788620065.878878 punch from 0x01333b to 0x01d9d3 (0xa699 bytes) 000179[0] 1788620065.939179 trunc from 0x851eed to 0x3dcf72 (0x474f7a bytes) 000180[0] 1788620066.007169 mapwrite 0x4fe81f thru 0x501089 (0x286b bytes) 000181[0] 1788620066.163191 mapread 0x156bed thru 0x15f15b (0x856f bytes) 000182[1] 1788620066.250334 write 0x0d0071 thru 0x0d672e (0x66be bytes) 000183[1] 1788620066.296645 trunc from 0x50108a to 0x9237d2 (0x422749 bytes) 000185[1] 1788620066.376847 read 0x399d31 thru 0x3a81be (0xe48e bytes) 000186[0] 1788620066.416735 write 0x6d5887 thru 0x6dfee9 (0xa663 bytes) 000187[1] 1788620066.458589 mapwrite 0x50614f thru 0x509c0a (0x3abc bytes) 000188[1] 1788620066.553549 mapread 0x90893f thru 0x909f96 (0x1658 bytes) 000189[0] 1788620066.568582 mapwrite 0x310e28 thru 0x31c122 (0xb2fb bytes) 000192[0] 1788620066.782685 trunc from 0x9237d3 to 0x4cc8f2 (0x456ee0 bytes) 000194[1] 1788620066.868062 punch from 0x453218 to 0x45b5b5 (0x839e bytes) 000197[1] 1788620066.955468 write 0x379b44 thru 0x3841f5 (0xa6b2 bytes) 000198[0] 1788620067.004425 mapread 0x3f9959 thru 0x407612 (0xdcba bytes) 000199[0] 1788620067.195401 write 0x4ada8f thru 0x4b36cd (0x5c3f bytes) 000200[1] 1788620067.248309 mapwrite 0x49ffd6 thru 0x4aa677 (0xa6a2 bytes) 000202[0] 1788620067.584068 read 0x3ee7e5 thru 0x3eed92 (0x5ae bytes) 000203[1] 1788620067.638046 mapread 0x1d4ca9 thru 0x1d4dbf (0x117 bytes) 000204[0] 1788620067.659525 read 0x2ab54b thru 0x2b7da6 (0xc85c bytes) 000205[1] 1788620067.739606 read 0x09fb1f thru 0x0ac2f5 (0xc7d7 bytes) 000208[1] 1788620067.849377 write 0x0c4694 thru 0x0c4f62 (0x8cf bytes) 000209[0] 1788620067.882762 read 0x3979b7 thru 0x39b1b7 (0x3801 bytes) 000210[1] 1788620067.971675 write 0x832542 thru 0x83680d (0x42cc bytes) 000211[1] 1788620068.051111 read 0x59b360 thru 0x5a0ec0 (0x5b61 bytes) 000212[1] 1788620068.101515 trunc from 0x83680e to 0x41a404 (0x41c409 bytes) 000214[1] 1788620068.259606 read 0x35d199 thru 0x35d205 (0x6d bytes) 000216[0] 1788620068.287596 mapread 0x181642 thru 0x18ecb4 (0xd673 bytes) 000217[0] 1788620068.408693 mapread 0x0bff7a thru 0x0ca13c (0xa1c3 bytes) 000220[0] 1788620068.527281 punch from 0x04fae4 to 0x04ff92 (0x4af bytes) 000221[0] 1788620068.565292 read 0x36979b thru 0x378737 (0xef9d bytes) 000222[0] 1788620068.596452 mapwrite 0x2150b1 thru 0x21b9ee (0x693e bytes) 000223[1] 1788620068.775254 read 0x0237b8 thru 0x032d02 (0xf54b bytes) 000224[1] 1788620068.825345 punch from 0x1dd79b to 0x1e1af0 (0x4356 bytes) 000225[1] 1788620068.845743 read 0x020cd0 thru 0x02b3e6 (0xa717 bytes) 000226[0] 1788620068.875403 read 0x35132f thru 0x35ce44 (0xbb16 bytes) 000227[0] 1788620068.916289 punch from 0x2e6f7a to 0x2ede60 (0x6ee7 bytes) 000228[0] 1788620068.967077 punch from 0x39bea5 to 0x3a6036 (0xa192 bytes) 000231[1] 1788620069.046547 read 0x15036e thru 0x1576ec (0x737f bytes) 000232[0] 1788620069.104010 mapread 0x340a50 thru 0x34495a (0x3f0b bytes) 000234[0] 1788620069.167033 punch from 0x20a424 to 0x20ee47 (0x4a24 bytes) 000235[1] 1788620069.236089 punch from 0x05956a to 0x05c23f (0x2cd6 bytes) 000236[0] 1788620069.272025 trunc from 0x41a405 to 0x8b6598 (0x49c194 bytes) 000237[0] 1788620069.390778 mapread 0x400392 thru 0x4086e0 (0x834f bytes) 000238[1] 1788620069.471592 read 0x28dc1b thru 0x29735c (0x9742 bytes) 000239[1] 1788620069.521175 read 0x31a2d7 thru 0x31e5b7 (0x42e1 bytes) 000240[0] 1788620069.585567 mapread 0x39c863 thru 0x3a78c3 (0xb061 bytes) 000241[0] 1788620069.663645 trunc from 0x8b6599 to 0x4c7333 (0x3ef265 bytes) 000242[1] 1788620069.753441 read 0x124de2 thru 0x12d808 (0x8a27 bytes) 000243[1] 1788620069.852158 mapread 0x281e2c thru 0x28cf38 (0xb10d bytes) 000245[1] 1788620069.924021 punch from 0x2ef722 to 0x2ff34c (0xfc2b bytes) 000247[0] 1788620070.025684 mapwrite 0x5bce89 thru 0x5c8520 (0xb698 bytes) 000248[0] 1788620070.261452 mapwrite 0x73ec4c thru 0x746db7 (0x816c bytes) 000249[0] 1788620070.381333 mapwrite 0x4384f2 thru 0x443144 (0xac53 bytes) 000250[1] 1788620070.532714 punch from 0x70ee0b to 0x713555 (0x474b bytes) 000251[1] 1788620070.588193 mapwrite 0x7d36a0 thru 0x7e32df (0xfc40 bytes) 000253[0] 1788620070.761181 mapwrite 0x224042 thru 0x225c62 (0x1c21 bytes) 000256[0] 1788620070.993229 write 0x5d6804 thru 0x5e1317 (0xab14 bytes) 000257[0] 1788620071.029577 mapwrite 0x48b907 thru 0x48fb8a (0x4284 bytes) 000258[0] 1788620071.134354 mapwrite 0x6f9d41 thru 0x7022fe (0x85be bytes) 000259[1] 1788620071.247784 trunc from 0x7e32e0 to 0x8f9e56 (0x116b77 bytes) 000262[1] 1788620071.403080 punch from 0x402b81 to 0x40d7c2 (0xac42 bytes) 000264[0] 1788620071.425225 read 0x4f17ae thru 0x4f9238 (0x7a8b bytes) 000266[1] 1788620071.513570 read 0x328b30 thru 0x32f5fd (0x6ace bytes) 000267[1] 1788620071.538198 punch from 0x283ef8 to 0x28be21 (0x7f2a bytes) 000270[0] 1788620071.590715 read 0x066e5c thru 0x075f2b (0xf0d0 bytes) 000271[1] 1788620071.613640 mapread 0x8cacf1 thru 0x8cb0a6 (0x3b6 bytes) 000272[1] 1788620071.623366 mapread 0x58e2eb thru 0x592b78 (0x488e bytes) 000273[1] 1788620071.648457 punch from 0x804e14 to 0x80df89 (0x9176 bytes) 000276[0] 1788620071.671725 write 0x74eb09 thru 0x7520ac (0x35a4 bytes) 000277[1] 1788620071.735116 write 0x644183 thru 0x644d4b (0xbc9 bytes) 000279[0] 1788620071.822001 mapwrite 0x7cc6ca thru 0x7d5950 (0x9287 bytes) 000280[0] 1788620071.962250 write 0x36e1ee thru 0x374db9 (0x6bcc bytes) 000281[1] 1788620071.995364 trunc from 0x8f9e57 to 0x4dcc4a (0x41d20c bytes) 000282[1] 1788620072.058218 mapwrite 0x9cd3c7 thru 0x9d2932 (0x556c bytes) 000283[1] 1788620072.182188 write 0x393ccd thru 0x3a2b10 (0xee44 bytes) 000284[1] 1788620072.275377 read 0x6c18f6 thru 0x6d0980 (0xf08b bytes) 000287[0] 1788620072.379046 mapread 0x6e5ea5 thru 0x6ebb72 (0x5cce bytes) 000289[0] 1788620072.489491 write 0x0901c0 thru 0x09918b (0x8fcc bytes) 000290[1] 1788620072.552484 write 0x42c0c5 thru 0x438486 (0xc3c2 bytes) 000291[0] 1788620072.609553 trunc from 0x9d2933 to 0x850d46 (0x181bec bytes) 000292[1] 1788620072.695992 mapwrite 0x030d13 thru 0x039861 (0x8b4f bytes) 000294[1] 1788620072.882569 read 0x1dc6c2 thru 0x1eb4da (0xee19 bytes) 000295[0] 1788620072.908216 read 0x5fa1f8 thru 0x603fd2 (0x9ddb bytes) 000298[1] 1788620072.979713 read 0x35308f thru 0x35b12a (0x809c bytes) 000299[0] 1788620073.010589 mapread 0x75c6b2 thru 0x76bba8 (0xf4f7 bytes) 000300[1] 1788620073.098151 read 0x84f984 thru 0x850d46 (0x13c3 bytes) 000301[1] 1788620073.140099 write 0x549b02 thru 0x549ca4 (0x1a3 bytes) 000303[1] 1788620073.193357 read 0x4732e4 thru 0x47ec75 (0xb992 bytes) 000305[1] 1788620073.265122 mapread 0x7e08ec thru 0x7e22e0 (0x19f5 bytes) 000306[0] 1788620073.323723 punch from 0x7f90aa to 0x7fd773 (0x46ca bytes) 000309[1] 1788620073.399506 read 0x6f18a7 thru 0x6ffd73 (0xe4cd bytes) 000310[1] 1788620073.425949 mapread 0x426ec6 thru 0x42eaa8 (0x7be3 bytes) 000311[1] 1788620073.525243 read 0x1d66b1 thru 0x1e55cb (0xef1b bytes) 000314[1] 1788620073.650389 write 0x379252 thru 0x388bb7 (0xf966 bytes) 000315[0] 1788620073.709961 read 0x185375 thru 0x18f89f (0xa52b bytes) truncating to largest ever: 0x9d9e2a 000317[0] 1788620073.755550 trunc from 0x850d47 to 0x9d9e29 (0x1890e3 bytes) 000318[1] 1788620073.813671 punch from 0x9b56d9 to 0x9bdb2c (0x8454 bytes) 000321[1] 1788620073.883769 mapwrite 0x810e5a thru 0x818b86 (0x7d2d bytes) 000323[0] 1788620074.022885 mapwrite 0x3d5172 thru 0x3d8c98 (0x3b27 bytes) 000324[1] 1788620074.213517 mapwrite 0x761356 thru 0x768719 (0x73c4 bytes) 000326[0] 1788620074.405411 punch from 0x008c5e to 0x016624 (0xd9c7 bytes) 000328[1] 1788620074.440380 write 0x604ab6 thru 0x61253f (0xda8a bytes) 000329[1] 1788620074.463660 write 0x9aafd6 thru 0x9b7716 (0xc741 bytes) 000332[0] 1788620074.568801 mapwrite 0x678802 thru 0x67ebef (0x63ee bytes) 000333[0] 1788620074.704015 punch from 0x142b0a to 0x14649b (0x3992 bytes) 000334[0] 1788620074.739128 write 0x21e5b4 thru 0x2212df (0x2d2c bytes) 000335[0] 1788620074.766488 punch from 0x8e6263 to 0x8f485a (0xe5f8 bytes) 000336[0] 1788620074.772463 trunc from 0x9d9e2a to 0x0d31dd (0x906c4c bytes) truncating to largest ever: 0x9f27c8 000338[0] 1788620074.850716 trunc from 0x0d31de to 0x9f27c7 (0x91f5ea bytes) 000339[0] 1788620074.912634 punch from 0x4d221a to 0x4d60da (0x3ec1 bytes) 000340[0] 1788620074.936588 read 0x4b4cbd thru 0x4c3e79 (0xf1bd bytes) 000341[0] 1788620074.990773 read 0x291f05 thru 0x29be72 (0x9f6e bytes) 000343[1] 1788620074.999796 read 0x63aeba thru 0x645fae (0xb0f5 bytes) 000344[1] 1788620075.048972 mapwrite 0x9bed2b thru 0x9c51d3 (0x64a9 bytes) 000345[0] 1788620075.202776 trunc from 0x9f27c8 to 0x7bace7 (0x237ae0 bytes) 000346[1] 1788620075.347353 mapwrite 0x1666b3 thru 0x168e2a (0x2778 bytes) 000348[1] 1788620075.560481 mapwrite 0x75d6c3 thru 0x76a508 (0xce46 bytes) 000350[1] 1788620075.726315 mapwrite 0x068f0d thru 0x075402 (0xc4f6 bytes) 000351[1] 1788620075.817492 write 0x0c37ee thru 0x0c6778 (0x2f8b bytes) 000352[1] 1788620075.848152 write 0x8cbff2 thru 0x8d1897 (0x58a6 bytes) 000354[0] 1788620075.886454 trunc from 0x8d1898 to 0x6a5768 (0x22c12f bytes) 000358[1] 1788620076.013582 read 0x2a3985 thru 0x2a3a42 (0xbe bytes) 000360[0] 1788620076.081449 mapread 0x1aca58 thru 0x1b77af (0xad58 bytes) 000361[0] 1788620076.130337 read 0x32a444 thru 0x32cb89 (0x2746 bytes) 000362[0] 1788620076.150264 punch from 0x1dd496 to 0x1e342c (0x5f97 bytes) 000363[0] 1788620076.204025 read 0x67b729 thru 0x681244 (0x5b1c bytes) 000366[0] 1788620076.281149 mapwrite 0x5c50d4 thru 0x5d1617 (0xc544 bytes) 000367[1] 1788620076.379536 read 0x08d983 thru 0x08fbf5 (0x2273 bytes) 000368[1] 1788620076.421650 mapread 0x597cb3 thru 0x598c4b (0xf99 bytes) 000369[0] 1788620076.452288 mapread 0x264844 thru 0x272940 (0xe0fd bytes) 000371[0] 1788620076.490777 mapread 0x625405 thru 0x62df39 (0x8b35 bytes) 000376[0] 1788620076.553757 mapwrite 0x635d37 thru 0x63cae1 (0x6dab bytes) 000378[1] 1788620076.755423 mapwrite 0x243a3c thru 0x246b14 (0x30d9 bytes) 000379[1] 1788620076.935121 read 0x0f7a18 thru 0x0fb601 (0x3bea bytes) 000384[1] 1788620077.075185 mapread 0x3862aa thru 0x394904 (0xe65b bytes) 000385[0] 1788620077.152388 punch from 0x44e69a to 0x44fd66 (0x16cd bytes) 000386[0] 1788620077.174425 write 0x522efe thru 0x52f310 (0xc413 bytes) 000388[0] 1788620077.231021 mapwrite 0x956890 thru 0x9601a5 (0x9916 bytes) 000389[1] 1788620077.380026 mapread 0x5918bd thru 0x5950d0 (0x3814 bytes) 000390[1] 1788620077.478609 mapread 0x47a7f4 thru 0x487372 (0xcb7f bytes) 000392[1] 1788620077.565587 write 0x7681ab thru 0x77773c (0xf592 bytes) 000394[0] 1788620077.642521 punch from 0x694624 to 0x696370 (0x1d4d bytes) 000396[1] 1788620077.665443 read 0x305b68 thru 0x308a04 (0x2e9d bytes) 000397[0] 1788620077.698471 write 0x859fe0 thru 0x85dd5f (0x3d80 bytes) 000398[1] 1788620077.766539 read 0x672c47 thru 0x67ab06 (0x7ec0 bytes) 000399[1] 1788620077.809582 mapwrite 0x4d2c6d thru 0x4db4cd (0x8861 bytes) 000401[0] 1788620077.936330 mapwrite 0x102b39 thru 0x10d1ae (0xa676 bytes) 000402[1] 1788620078.094261 read 0x070a08 thru 0x079833 (0x8e2c bytes) 000404[1] 1788620078.146697 trunc from 0x9601a6 to 0x15421b (0x80bf8a bytes) 000406[1] 1788620078.217469 mapread 0x08aa91 thru 0x09129f (0x680f bytes) 000407[1] 1788620078.297576 punch from 0x0b29c9 to 0x0b7089 (0x46c1 bytes) 000410[1] 1788620078.306715 trunc from 0x15421c to 0x80e18e (0x6b9f73 bytes) 000411[1] 1788620078.365585 mapwrite 0x9311dd thru 0x93ab16 (0x993a bytes) 000414[0] 1788620078.519523 punch from 0x1850dc to 0x19062e (0xb553 bytes) 000416[1] 1788620078.588257 trunc from 0x93ab17 to 0x318bdc (0x621f3a bytes) 000417[0] 1788620078.650398 trunc from 0x318bdd to 0x1dd271 (0x13b96b bytes) 000420[0] 1788620078.746780 punch from 0x1b248d to 0x1b627c (0x3df0 bytes) 000421[0] 1788620078.785523 punch from 0x021871 to 0x0311de (0xf96e bytes) 000423[0] 1788620078.818767 punch from 0x1b8fe6 to 0x1c2918 (0x9933 bytes) 000425[0] 1788620078.882441 trunc from 0x1dd272 to 0x561448 (0x3841d7 bytes) 000430[0] 1788620078.951953 mapwrite 0x925b64 thru 0x9295bd (0x3a5a bytes) 000431[0] 1788620079.057957 mapwrite 0x5793c6 thru 0x57b688 (0x22c3 bytes) 000433[0] 1788620079.149945 mapread 0x298e75 thru 0x29e587 (0x5713 bytes) 000435[0] 1788620079.198272 write 0x001a5d thru 0x0059c6 (0x3f6a bytes) 000436[1] 1788620079.211095 read 0x388887 thru 0x38f460 (0x6bda bytes) 000437[1] 1788620079.260953 mapwrite 0x4498dc thru 0x45401a (0xa73f bytes) 000438[1] 1788620079.586945 write 0x7139f5 thru 0x7169fe (0x300a bytes) 000439[1] 1788620079.605613 write 0x0f4a30 thru 0x0fc2b5 (0x7886 bytes) 000440[0] 1788620079.686465 write 0x1a5a22 thru 0x1a7734 (0x1d13 bytes) 000441[1] 1788620079.757642 mapread 0x43ad28 thru 0x43cbc7 (0x1ea0 bytes) 000443[0] 1788620079.825206 trunc from 0x9295be to 0x1cc8ef (0x75ccce bytes) 000448[1] 1788620079.988112 punch from 0x008fc3 to 0x00abb4 (0x1bf2 bytes) 000449[1] 1788620080.024234 write 0x07f3e2 thru 0x0882f1 (0x8f10 bytes) 000450[0] 1788620080.058981 punch from 0x064627 to 0x066993 (0x236d bytes) 000451[0] 1788620080.098600 trunc from 0x1cc8f0 to 0x09c384 (0x13056b bytes) 000457[1] 1788620080.303659 mapread 0x0674c6 thru 0x06b862 (0x439d bytes) 000459[1] 1788620080.376897 trunc from 0x09c385 to 0x3f0e1d (0x354a99 bytes) 000460[0] 1788620080.436757 punch from 0x11c73d to 0x12985f (0xd123 bytes) 000461[0] 1788620080.478620 trunc from 0x3f0e1e to 0x42c376 (0x3b559 bytes) 000464[1] 1788620080.589591 read 0x11445d thru 0x119e1d (0x59c1 bytes) 000466[0] 1788620080.675650 punch from 0x3ac0f4 to 0x3b66bf (0xa5cc bytes) 000469[1] 1788620080.745509 mapread 0x0e6f11 thru 0x0f2022 (0xb112 bytes) 000473[1] 1788620080.815059 punch from 0x13821f to 0x142de6 (0xabc8 bytes) 000477[0] 1788620080.914229 read 0x1373b3 thru 0x14560d (0xe25b bytes) 000478[1] 1788620080.963777 write 0x506317 thru 0x511777 (0xb461 bytes) 000480[1] 1788620081.009976 mapwrite 0x48217c thru 0x488946 (0x67cb bytes) 000481[1] 1788620081.216294 mapwrite 0x5450ea thru 0x54e2a0 (0x91b7 bytes) 000484[1] 1788620081.359597 punch from 0x502616 to 0x505bc3 (0x35ae bytes) 000485[1] 1788620081.386206 write 0x952b6a thru 0x95a8c7 (0x7d5e bytes) 000487[1] 1788620081.413581 read 0x5299d6 thru 0x5370e3 (0xd70e bytes) 000488[1] 1788620081.448184 write 0x466871 thru 0x473dfe (0xd58e bytes) 000489[1] 1788620081.449663 read 0x7c70a7 thru 0x7cd074 (0x5fce bytes) 000491[0] 1788620081.518434 trunc from 0x95a8c8 to 0x583852 (0x3d7075 bytes) 000492[0] 1788620081.615881 trunc from 0x583853 to 0x5dbc28 (0x583d6 bytes) 000494[0] 1788620081.709490 trunc from 0x5dbc29 to 0x286e1c (0x354e0c bytes) 000496[0] 1788620081.833951 read 0x19cd05 thru 0x1a6f0b (0xa207 bytes) 000497[0] 1788620081.901386 mapwrite 0x238171 thru 0x238769 (0x5f9 bytes) 000498[1] 1788620081.985341 write 0x4c6d38 thru 0x4ce81c (0x7ae5 bytes) 000499[0] 1788620082.013618 punch from 0x1aa16f to 0x1ad508 (0x339a bytes) 000500[0] 1788620082.034573 punch from 0x28fda4 to 0x299b34 (0x9d91 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 82 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000006[0] 1788620082.920251 mapwrite 0x5be000 thru 0x5c6d36 (0x8d37 bytes) 000008[0] 1788620083.107468 punch from 0x1cf74b to 0x1ddc35 (0xe4eb bytes) truncating to largest ever: 0x96f7a5 000009[1] 1788620083.128753 trunc from 0x5c6d37 to 0x96f7a4 (0x3a8a6e bytes) 000011[1] 1788620083.201424 mapwrite 0x1bd000 thru 0x1c821a (0xb21b bytes) 000012[1] 1788620083.348725 punch from 0x5c4022 to 0x5d3212 (0xf1f1 bytes) 000014[1] 1788620083.426086 read 0x7f6000 thru 0x8050e6 (0xf0e7 bytes) 000015[1] 1788620083.461472 punch from 0x1ee8dd to 0x1fa65f (0xbd83 bytes) 000016[0] 1788620083.507169 mapread 0x3f2000 thru 0x3fded1 (0xbed2 bytes) 000017[1] 1788620083.601475 mapread 0x0f0000 thru 0x0fa44d (0xa44e bytes) 000019[0] 1788620083.653973 mapwrite 0x431000 thru 0x43c001 (0xb002 bytes) 000020[0] 1788620083.787561 trunc from 0x96f7a5 to 0x64bdd6 (0x3239ce bytes) 000021[1] 1788620083.810941 punch from 0x02fd29 to 0x034e67 (0x513f bytes) 000023[1] 1788620083.851729 mapread 0x5d5000 thru 0x5ddd80 (0x8d81 bytes) 000024[1] 1788620083.885467 mapwrite 0x9ab000 thru 0x9ad79b (0x279c bytes) 000025[0] 1788620084.039312 write_OD 0x84c000 thru 0x857fff (0xc000 bytes) 000026[1] 1788620084.149665 write 0x11b000 thru 0x1235aa (0x85ab bytes) 000027[0] 1788620084.228941 write_OD 0x72c000 thru 0x72ffff (0x4000 bytes) 000030[1] 1788620084.334713 mapread 0x86e000 thru 0x86ee12 (0xe13 bytes) 000031[0] 1788620084.344233 read_OD 0x771000 thru 0x77afff (0xa000 bytes) 000035[0] 1788620084.353374 write_OD 0x682000 thru 0x687fff (0x6000 bytes) 000036[0] 1788620084.407646 mapread 0x78c000 thru 0x795c31 (0x9c32 bytes) 000037[0] 1788620084.430267 trunc from 0x9ad79c to 0x39383b (0x619f60 bytes) 000039[0] 1788620084.471327 read_OD 0x329000 thru 0x32cfff (0x4000 bytes) 000040[1] 1788620084.501883 mapread 0x154000 thru 0x15a9f7 (0x69f8 bytes) 000041[1] 1788620084.538995 write 0x6d4000 thru 0x6e187c (0xd87d bytes) 000042[1] 1788620084.608643 mapwrite 0x870000 thru 0x8785c9 (0x85ca bytes) 000043[0] 1788620084.818394 punch from 0x736e08 to 0x73f8a9 (0x8aa2 bytes) 000047[0] 1788620084.874784 write_OD 0x0c3000 thru 0x0cbfff (0x9000 bytes) 000049[1] 1788620084.951271 mapread 0x494000 thru 0x499d1b (0x5d1c bytes) 000050[1] 1788620084.979602 read 0x49b000 thru 0x4a5ca2 (0xaca3 bytes) 000051[0] 1788620084.989231 mapwrite 0x0e5000 thru 0x0e836d (0x336e bytes) 000054[0] 1788620085.143132 punch from 0x862d40 to 0x86427f (0x1540 bytes) 000055[1] 1788620085.174329 mapread 0x479000 thru 0x480057 (0x7058 bytes) 000059[1] 1788620085.220311 punch from 0x530a35 to 0x534bfb (0x41c7 bytes) 000061[0] 1788620085.278483 punch from 0x178a43 to 0x180e73 (0x8431 bytes) 000062[1] 1788620085.289128 read 0x3a3000 thru 0x3a67fe (0x37ff bytes) 000063[1] 1788620085.310960 punch from 0x6b395d to 0x6c2324 (0xe9c8 bytes) 000065[0] 1788620085.343050 mapwrite 0x086000 thru 0x093e5d (0xde5e bytes) 000066[1] 1788620085.579364 punch from 0x28a5f8 to 0x29a1d8 (0xfbe1 bytes) 000067[1] 1788620085.598583 write 0x5a2000 thru 0x5a49ac (0x29ad bytes) 000068[1] 1788620085.619555 read 0x08f000 thru 0x09a68c (0xb68d bytes) 000072[0] 1788620085.655759 trunc from 0x8785ca to 0x53ab4e (0x33da7b bytes) 000073[1] 1788620085.708257 mapwrite 0x4fa000 thru 0x508862 (0xe863 bytes) 000074[1] 1788620085.809070 punch from 0x3c8355 to 0x3cef90 (0x6c3c bytes) 000075[1] 1788620085.821296 mapwrite 0x72e000 thru 0x731b85 (0x3b86 bytes) 000079[0] 1788620086.013292 mapread 0x09b000 thru 0x0a45a3 (0x95a4 bytes) 000081[0] 1788620086.112696 mapwrite 0x618000 thru 0x625c6a (0xdc6b bytes) 000082[1] 1788620086.297236 mapwrite 0x423000 thru 0x423bd1 (0xbd2 bytes) 000083[0] 1788620086.405245 write_OD 0x544000 thru 0x545fff (0x2000 bytes) 000084[0] 1788620086.492395 mapread 0x056000 thru 0x05b69f (0x56a0 bytes) 000085[1] 1788620086.534438 write 0x082000 thru 0x08d1e0 (0xb1e1 bytes) 000087[1] 1788620086.576120 punch from 0x0c6c1f to 0x0c9986 (0x2d68 bytes) 000091[1] 1788620086.630222 punch from 0x0e7fd1 to 0x0f4555 (0xc585 bytes) 000092[0] 1788620086.652528 trunc from 0x731b86 to 0x216ba7 (0x51afde bytes) 000093[0] 1788620086.690778 trunc from 0x216ba8 to 0x6e1436 (0x4ca88f bytes) 000094[1] 1788620086.724259 punch from 0x633563 to 0x63afbd (0x7a5b bytes) 000095[0] 1788620086.788727 write_OD 0x98e000 thru 0x98ffff (0x2000 bytes) 000096[1] 1788620086.881108 write 0x54b000 thru 0x5514cf (0x64d0 bytes) 000097[0] 1788620086.923762 mapread 0x06e000 thru 0x06f58b (0x158c bytes) 000098[0] 1788620086.974479 read_OD 0x889000 thru 0x889fff (0x1000 bytes) 000099[0] 1788620086.985720 mapread 0x624000 thru 0x62563d (0x163e bytes) 000100[1] 1788620086.997453 punch from 0x200f21 to 0x203164 (0x2244 bytes) 000101[1] 1788620087.015434 write 0x86b000 thru 0x8717e8 (0x67e9 bytes) 000102[0] 1788620087.034751 punch from 0x465ea0 to 0x46db31 (0x7c92 bytes) 000103[1] 1788620087.060341 trunc from 0x990000 to 0x1706ab (0x81f954 bytes) 000105[1] 1788620087.108370 mapread 0x05f000 thru 0x062c91 (0x3c92 bytes) 000108[1] 1788620087.215719 trunc from 0x1706ac to 0x625d89 (0x4b56de bytes) 000110[1] 1788620087.277452 punch from 0x5a5138 to 0x5ae93e (0x9807 bytes) 000111[1] 1788620087.302289 mapread 0x5bf000 thru 0x5cd141 (0xe142 bytes) 000112[0] 1788620087.356447 mapread 0x035000 thru 0x03c10c (0x710d bytes) 000113[1] 1788620087.419775 mapwrite 0x4f2000 thru 0x4f2c6a (0xc6b bytes) 000114[0] 1788620087.556745 mapread 0x471000 thru 0x47b8e2 (0xa8e3 bytes) 000116[1] 1788620087.668076 read 0x5ee000 thru 0x5f0380 (0x2381 bytes) 000117[0] 1788620087.677724 mapread 0x3f4000 thru 0x403480 (0xf481 bytes) 000119[1] 1788620087.706033 trunc from 0x625d8a to 0x7842d8 (0x15e54f bytes) 000120[0] 1788620087.751502 read_OD 0x6d9000 thru 0x6e3fff (0xb000 bytes) 000121[0] 1788620087.805453 mapwrite 0x5ee000 thru 0x5fc90f (0xe910 bytes) 000122[0] 1788620087.923020 trunc from 0x7842d9 to 0x8eab8d (0x1668b5 bytes) 000123[1] 1788620087.973565 read 0x050000 thru 0x055d4b (0x5d4c bytes) 000124[1] 1788620088.024626 trunc from 0x8eab8e to 0x05fbe4 (0x88afa9 bytes) 000125[0] 1788620088.061506 trunc from 0x05fbe5 to 0x36ea6b (0x30ee87 bytes) 000128[1] 1788620088.143374 write 0x36e000 thru 0x370784 (0x2785 bytes) 000129[0] 1788620088.157137 read_OD 0x138000 thru 0x143fff (0xc000 bytes) 000130[0] 1788620088.192049 punch from 0x0c60b3 to 0x0cabc7 (0x4b15 bytes) 000133[1] 1788620088.292619 punch from 0x201aaf to 0x203e55 (0x23a7 bytes) 000134[0] 1788620088.349106 write_OD 0x844000 thru 0x84dfff (0xa000 bytes) 000136[0] 1788620088.472301 trunc from 0x84e000 to 0x7b0c84 (0x9d37b bytes) 000137[1] 1788620088.527058 trunc from 0x7b0c85 to 0x2d0a2d (0x4e0257 bytes) 000138[1] 1788620088.630274 mapread 0x06a000 thru 0x079f04 (0xff05 bytes) 000139[0] 1788620088.682223 punch from 0x146c86 to 0x14a4b2 (0x382d bytes) 000141[0] 1788620088.728685 trunc from 0x2d0a2e to 0x641ba6 (0x371179 bytes) 000142[0] 1788620088.767555 mapwrite 0x285000 thru 0x288c29 (0x3c2a bytes) 000143[0] 1788620088.971287 punch from 0x13f4e4 to 0x144387 (0x4ea4 bytes) 000146[1] 1788620089.028787 read 0x45e000 thru 0x46a7a7 (0xc7a8 bytes) 000147[1] 1788620089.078127 punch from 0x09bb4e to 0x0aaad1 (0xef84 bytes) 000149[1] 1788620089.166068 mapread 0x2b3000 thru 0x2bdce0 (0xace1 bytes) 000152[1] 1788620089.221711 punch from 0x48e9c5 to 0x49424f (0x588b bytes) 000153[1] 1788620089.228361 mapwrite 0x5a5000 thru 0x5aed43 (0x9d44 bytes) 000154[1] 1788620089.381246 mapread 0x441000 thru 0x44f892 (0xe893 bytes) 000156[1] 1788620089.474191 mapwrite 0x801000 thru 0x803f9a (0x2f9b bytes) 000159[0] 1788620089.647690 punch from 0x6a233d to 0x6a330b (0xfcf bytes) 000160[1] 1788620089.665611 punch from 0x34a999 to 0x358468 (0xdad0 bytes) 000165[0] 1788620089.699188 trunc from 0x803f9b to 0x65a005 (0x1a9f95 bytes) 000167[0] 1788620089.762117 write_OD 0x2fd000 thru 0x303fff (0x7000 bytes) truncating to largest ever: 0x99c77d 000168[1] 1788620089.863229 trunc from 0x65a006 to 0x99c77c (0x342777 bytes) truncating to largest ever: 0x9c4cdf 000169[0] 1788620089.970759 trunc from 0x99c77d to 0x9c4cde (0x28562 bytes) 000170[0] 1788620090.017526 write_OD 0x778000 thru 0x786fff (0xf000 bytes) 000171[0] 1788620090.113211 write_OD 0x90b000 thru 0x90ffff (0x5000 bytes) 000172[1] 1788620090.242599 mapwrite 0x200000 thru 0x20ca56 (0xca57 bytes) 000173[1] 1788620090.543740 trunc from 0x9c4cdf to 0x1705a5 (0x854739 bytes) 000174[1] 1788620090.612260 punch from 0x13044d to 0x138035 (0x7be9 bytes) 000175[1] 1788620090.640271 mapwrite 0x530000 thru 0x5358d2 (0x58d3 bytes) 000176[0] 1788620090.772322 trunc from 0x5358d3 to 0x2c6ddb (0x26eaf7 bytes) 000177[0] 1788620090.817351 mapwrite 0x31e000 thru 0x322d1c (0x4d1d bytes) 000179[0] 1788620090.940369 mapwrite 0x855000 thru 0x861674 (0xc675 bytes) 000180[0] 1788620091.202644 read_OD 0x231000 thru 0x232fff (0x2000 bytes) 000182[0] 1788620091.255253 read_OD 0x238000 thru 0x23dfff (0x6000 bytes) 000185[1] 1788620091.278262 write 0x268000 thru 0x272e0c (0xae0d bytes) 000187[1] 1788620091.331181 read 0x638000 thru 0x63df1e (0x5f1f bytes) 000191[1] 1788620091.445647 read 0x5b4000 thru 0x5b97c5 (0x57c6 bytes) 000194[1] 1788620091.565493 mapread 0x133000 thru 0x13333c (0x33d bytes) 000195[1] 1788620091.611132 read 0x43a000 thru 0x43e65e (0x465f bytes) 000197[1] 1788620091.735462 read 0x27f000 thru 0x28532b (0x632c bytes) 000199[1] 1788620091.807380 mapread 0x5ff000 thru 0x60e1c0 (0xf1c1 bytes) 000200[1] 1788620091.851488 trunc from 0x861675 to 0x8118f9 (0x4fd7b bytes) 000201[1] 1788620091.890493 mapwrite 0x04b000 thru 0x052ce1 (0x7ce2 bytes) 000202[1] 1788620092.089418 trunc from 0x8118fa to 0x2296b7 (0x5e8242 bytes) 000203[0] 1788620092.124696 mapwrite 0x00d000 thru 0x00f9b1 (0x29b2 bytes) 000204[1] 1788620092.301032 read 0x013000 thru 0x01f654 (0xc655 bytes) 000205[0] 1788620092.393517 read_OD 0x15a000 thru 0x165fff (0xc000 bytes) 000206[0] 1788620092.428221 punch from 0x05348a to 0x062fd6 (0xfb4d bytes) 000208[0] 1788620092.472343 mapread 0x036000 thru 0x04020c (0xa20d bytes) 000210[1] 1788620092.552156 trunc from 0x2296b8 to 0x3e8486 (0x1bedcf bytes) 000211[1] 1788620092.610686 trunc from 0x3e8487 to 0x2c3c0a (0x12487c bytes) 000212[0] 1788620092.683274 mapwrite 0x229000 thru 0x23026f (0x7270 bytes) 000216[1] 1788620092.974355 mapwrite 0x114000 thru 0x11e67b (0xa67c bytes) 000217[1] 1788620093.365102 mapwrite 0x47e000 thru 0x4821df (0x41e0 bytes) 000222[0] 1788620093.537183 read_OD 0x071000 thru 0x076fff (0x6000 bytes) 000224[1] 1788620093.569306 write 0x09d000 thru 0x09e7a1 (0x17a2 bytes) 000228[0] 1788620093.633242 write_OD 0x9c7000 thru 0x9d2fff (0xc000 bytes) 000232[0] 1788620093.711953 write_OD 0x402000 thru 0x40cfff (0xb000 bytes) 000233[1] 1788620093.759529 read 0x307000 thru 0x30df74 (0x6f75 bytes) 000234[0] 1788620093.773632 write_OD 0x87b000 thru 0x87dfff (0x3000 bytes) 000235[0] 1788620093.824345 read_OD 0x568000 thru 0x575fff (0xe000 bytes) 000238[0] 1788620093.836732 write_OD 0x237000 thru 0x23afff (0x4000 bytes) 000239[0] 1788620093.887583 mapread 0x84e000 thru 0x85d60f (0xf610 bytes) 000243[1] 1788620093.924179 trunc from 0x9d3000 to 0x638a99 (0x39a566 bytes) truncating to largest ever: 0x9f323f 000244[1] 1788620093.970323 trunc from 0x638a9a to 0x9f323e (0x3ba7a5 bytes) 000245[0] 1788620094.051353 punch from 0x10ff60 to 0x11173f (0x17e0 bytes) 000246[0] 1788620094.086492 write_OD 0x2cb000 thru 0x2cffff (0x5000 bytes) 000247[0] 1788620094.223205 read_OD 0x085000 thru 0x08dfff (0x9000 bytes) 000248[1] 1788620094.246314 mapread 0x7d3000 thru 0x7dbcac (0x8cad bytes) 000250[0] 1788620094.292389 trunc from 0x9f323f to 0x61065f (0x3e2bdf bytes) 000251[1] 1788620094.353798 read 0x174000 thru 0x174879 (0x87a bytes) 000253[0] 1788620094.414884 mapread 0x2b3000 thru 0x2bfdf7 (0xcdf8 bytes) 000254[0] 1788620094.475396 mapwrite 0x41e000 thru 0x42b4b8 (0xd4b9 bytes) 000255[0] 1788620094.662771 mapwrite 0x68e000 thru 0x698bb3 (0xabb4 bytes) 000260[0] 1788620095.016077 write_OD 0x9d1000 thru 0x9d9fff (0x9000 bytes) 000261[1] 1788620095.085282 write 0x6ea000 thru 0x6f566b (0xb66c bytes) 000264[0] 1788620095.153801 mapwrite 0x96a000 thru 0x971b46 (0x7b47 bytes) 000266[0] 1788620095.276761 write_OD 0x0b5000 thru 0x0bafff (0x6000 bytes) 000269[0] 1788620095.377127 mapread 0x819000 thru 0x8191a7 (0x1a8 bytes) 000270[0] 1788620095.392233 punch from 0x322fd3 to 0x327a82 (0x4ab0 bytes) 000271[1] 1788620095.425967 mapwrite 0x372000 thru 0x380ce7 (0xece8 bytes) 000272[0] 1788620095.616272 mapwrite 0x27e000 thru 0x281c2a (0x3c2b bytes) 000274[0] 1788620095.781748 mapwrite 0x8b6000 thru 0x8c0cdc (0xacdd bytes) 000275[1] 1788620095.881236 read 0x1bd000 thru 0x1c868e (0xb68f bytes) 000278[1] 1788620095.925281 mapread 0x104000 thru 0x10baf3 (0x7af4 bytes) 000279[1] 1788620095.960941 read 0x926000 thru 0x930d06 (0xad07 bytes) 000280[1] 1788620095.969767 read 0x6bc000 thru 0x6bdeeb (0x1eec bytes) 000281[0] 1788620095.976447 mapread 0x77a000 thru 0x7863d8 (0xc3d9 bytes) 000282[1] 1788620096.015616 trunc from 0x9da000 to 0x0160c2 (0x9c3f3d bytes) 000284[1] 1788620096.076526 read 0x00f000 thru 0x011df3 (0x2df4 bytes) 000286[0] 1788620096.121606 read_OD 0x001000 thru 0x00afff (0xa000 bytes) 000287[1] 1788620096.163235 read 0x003000 thru 0x008dcb (0x5dcc bytes) 000289[0] 1788620096.204255 mapread 0x009000 thru 0x015b10 (0xcb11 bytes) 000290[0] 1788620096.225801 write_OD 0x067000 thru 0x069fff (0x3000 bytes) 000291[1] 1788620096.294562 mapread 0x015000 thru 0x01ba39 (0x6a3a bytes) 000292[0] 1788620096.321232 read_OD 0x053000 thru 0x05ffff (0xd000 bytes) 000294[1] 1788620096.343363 read 0x030000 thru 0x035b8a (0x5b8b bytes) 000295[1] 1788620096.344110 mapread 0x05b000 thru 0x0637c5 (0x87c6 bytes) 000297[0] 1788620096.345940 write_OD 0x5a9000 thru 0x5abfff (0x3000 bytes) 000300[0] 1788620096.427168 mapwrite 0x671000 thru 0x674866 (0x3867 bytes) 000301[0] 1788620096.717691 punch from 0x3d2c23 to 0x3da407 (0x77e5 bytes) 000303[1] 1788620096.740242 write 0x095000 thru 0x099605 (0x4606 bytes) 000304[1] 1788620096.796456 write 0x5f4000 thru 0x5f6011 (0x2012 bytes) 000305[1] 1788620096.904992 punch from 0x54a366 to 0x550526 (0x61c1 bytes) 000307[0] 1788620096.932644 mapwrite 0x8f4000 thru 0x902c82 (0xec83 bytes) 000308[1] 1788620097.162733 write 0x0ac000 thru 0x0b7848 (0xb849 bytes) 000310[1] 1788620097.229420 write 0x3c0000 thru 0x3c8a17 (0x8a18 bytes) 000313[1] 1788620097.355541 write 0x59f000 thru 0x5a572f (0x6730 bytes) 000314[0] 1788620097.422596 read_OD 0x24b000 thru 0x24cfff (0x2000 bytes) 000315[1] 1788620097.476139 write 0x2ef000 thru 0x2f4ef0 (0x5ef1 bytes) 000319[1] 1788620097.536291 mapwrite 0x37a000 thru 0x37b6ea (0x16eb bytes) 000321[0] 1788620097.656260 write_OD 0x4ed000 thru 0x4fafff (0xe000 bytes) 000323[1] 1788620097.727074 mapread 0x0bb000 thru 0x0c83d9 (0xd3da bytes) 000324[0] 1788620097.815725 write_OD 0x073000 thru 0x07efff (0xc000 bytes) 000326[0] 1788620097.881311 punch from 0x618791 to 0x624917 (0xc187 bytes) 000327[1] 1788620097.948917 read 0x670000 thru 0x67143f (0x1440 bytes) 000328[1] 1788620097.978495 read 0x24a000 thru 0x25703a (0xd03b bytes) 000329[0] 1788620098.001898 write_OD 0x407000 thru 0x40afff (0x4000 bytes) 000330[1] 1788620098.097256 punch from 0x51fafc to 0x52d81f (0xdd24 bytes) 000331[0] 1788620098.122479 mapread 0x4da000 thru 0x4e8706 (0xe707 bytes) 000332[1] 1788620098.188969 write 0x0a2000 thru 0x0abc1b (0x9c1c bytes) 000333[0] 1788620098.235758 read_OD 0x7e4000 thru 0x7eafff (0x7000 bytes) 000335[1] 1788620098.302661 write 0x92a000 thru 0x92ba87 (0x1a88 bytes) 000336[0] 1788620098.314680 write_OD 0x144000 thru 0x146fff (0x3000 bytes) 000340[1] 1788620098.442135 mapread 0x8d2000 thru 0x8d2d6f (0xd70 bytes) 000342[0] 1788620098.450466 read_OD 0x5f7000 thru 0x5f8fff (0x2000 bytes) 000343[0] 1788620098.460351 mapread 0x407000 thru 0x415082 (0xe083 bytes) 000344[0] 1788620098.519799 mapwrite 0x5ec000 thru 0x5f03dd (0x43de bytes) 000345[0] 1788620098.610139 punch from 0x894e90 to 0x8974df (0x2650 bytes) 000347[1] 1788620098.637130 punch from 0x8119c7 to 0x81e9da (0xd014 bytes) 000348[0] 1788620098.673755 write_OD 0x6a9000 thru 0x6b2fff (0xa000 bytes) 000349[1] 1788620098.763849 mapwrite 0x9ef000 thru 0x9fb784 (0xc785 bytes) 000351[0] 1788620098.901312 read_OD 0x35f000 thru 0x36cfff (0xe000 bytes) 000352[0] 1788620098.931145 write_OD 0x5f7000 thru 0x5f8fff (0x2000 bytes) 000354[0] 1788620098.986259 mapwrite 0x542000 thru 0x54a5bd (0x85be bytes) 000356[0] 1788620099.071673 write_OD 0x767000 thru 0x767fff (0x1000 bytes) 000357[1] 1788620099.162388 read 0x213000 thru 0x21a1fb (0x71fc bytes) 000361[0] 1788620099.177182 read_OD 0x2e9000 thru 0x2edfff (0x5000 bytes) 000362[0] 1788620099.185892 mapwrite 0x122000 thru 0x128d19 (0x6d1a bytes) 000363[1] 1788620099.341877 write 0x89d000 thru 0x8a463e (0x763f bytes) 000364[1] 1788620099.401049 mapread 0x648000 thru 0x6520e1 (0xa0e2 bytes) 000368[0] 1788620099.503567 punch from 0x16003c to 0x16b504 (0xb4c9 bytes) 000369[1] 1788620099.531058 mapwrite 0x1d8000 thru 0x1d9d63 (0x1d64 bytes) 000371[1] 1788620099.642313 mapwrite 0x473000 thru 0x47b034 (0x8035 bytes) 000377[1] 1788620099.822382 mapwrite 0x0c5000 thru 0x0c817b (0x317c bytes) 000378[1] 1788620099.937803 trunc from 0x9fb785 to 0x5b5fda (0x4457aa bytes) 000379[0] 1788620100.007530 read_OD 0x383000 thru 0x38ffff (0xd000 bytes) 000380[1] 1788620100.080580 punch from 0x061eb5 to 0x0653f9 (0x3545 bytes) 000381[0] 1788620100.110556 mapread 0x29d000 thru 0x2a5f5c (0x8f5d bytes) 000384[0] 1788620100.246806 punch from 0x00e554 to 0x01683f (0x82ec bytes) 000385[1] 1788620100.284913 mapread 0x443000 thru 0x45118f (0xe190 bytes) 000388[1] 1788620100.388156 mapwrite 0x73a000 thru 0x7495e9 (0xf5ea bytes) 000389[0] 1788620100.507568 write_OD 0x523000 thru 0x52efff (0xc000 bytes) 000390[1] 1788620100.591553 mapread 0x17e000 thru 0x17f623 (0x1624 bytes) 000391[0] 1788620100.638155 punch from 0x5c9d03 to 0x5d0c2f (0x6f2d bytes) 000392[1] 1788620100.673009 write 0x600000 thru 0x607f73 (0x7f74 bytes) 000394[1] 1788620100.745259 mapread 0x5ef000 thru 0x5efeed (0xeee bytes) 000395[0] 1788620100.791807 read_OD 0x35d000 thru 0x363fff (0x7000 bytes) 000396[1] 1788620100.860163 mapread 0x6d0000 thru 0x6dee86 (0xee87 bytes) 000399[0] 1788620100.920923 punch from 0x5754d5 to 0x57834d (0x2e79 bytes) 000403[0] 1788620100.976019 mapread 0x133000 thru 0x140e22 (0xde23 bytes) 000404[0] 1788620101.022511 mapwrite 0x686000 thru 0x68c6a4 (0x66a5 bytes) 000405[1] 1788620101.119767 trunc from 0x7495ea to 0x727a39 (0x21bb0 bytes) 000406[1] 1788620101.163774 write 0x4c4000 thru 0x4cf43d (0xb43e bytes) 000407[0] 1788620101.215464 punch from 0x560702 to 0x568b61 (0x8460 bytes) 000409[1] 1788620101.303115 mapread 0x113000 thru 0x11f2f2 (0xc2f3 bytes) 000410[1] 1788620101.442631 write 0x54a000 thru 0x5539cc (0x99cd bytes) 000411[1] 1788620101.534436 punch from 0x66030b to 0x66d32d (0xd023 bytes) 000412[0] 1788620101.594474 read_OD 0x3b6000 thru 0x3bafff (0x5000 bytes) 000415[0] 1788620101.673062 read_OD 0x19e000 thru 0x1aafff (0xd000 bytes) 000416[0] 1788620101.691054 read_OD 0x2ad000 thru 0x2b1fff (0x5000 bytes) 000419[1] 1788620101.730167 punch from 0x68718e to 0x693d1b (0xcb8e bytes) 000421[1] 1788620101.839478 punch from 0x6ed463 to 0x6f3e80 (0x6a1e bytes) 000422[0] 1788620101.856639 mapread 0x608000 thru 0x60e36d (0x636e bytes) 000423[0] 1788620101.952500 mapread 0x63a000 thru 0x63bc02 (0x1c03 bytes) 000424[0] 1788620101.975262 punch from 0x12b491 to 0x135109 (0x9c79 bytes) 000426[1] 1788620102.105918 read 0x670000 thru 0x67270c (0x270d bytes) 000427[0] 1788620102.152224 read_OD 0x441000 thru 0x447fff (0x7000 bytes) 000428[1] 1788620102.192171 read 0x41b000 thru 0x41eca0 (0x3ca1 bytes) 000429[0] 1788620102.204581 write_OD 0x231000 thru 0x235fff (0x5000 bytes) 000430[1] 1788620102.341506 write 0x7f8000 thru 0x7fe442 (0x6443 bytes) 000432[0] 1788620102.401715 read_OD 0x076000 thru 0x082fff (0xd000 bytes) 000433[0] 1788620102.500888 write_OD 0x6e9000 thru 0x6f7fff (0xf000 bytes) 000437[0] 1788620102.737980 mapread 0x035000 thru 0x03f22f (0xa230 bytes) 000438[1] 1788620102.804235 write 0x4f1000 thru 0x4f58b5 (0x48b6 bytes) 000439[1] 1788620102.871595 mapread 0x7be000 thru 0x7c23d0 (0x43d1 bytes) 000441[1] 1788620102.926025 mapread 0x3d7000 thru 0x3e48d0 (0xd8d1 bytes) 000444[1] 1788620103.029641 punch from 0x3e9aff to 0x3f4c36 (0xb138 bytes) 000445[0] 1788620103.036121 write_OD 0x77f000 thru 0x789fff (0xb000 bytes) 000448[0] 1788620103.229117 mapread 0x0fd000 thru 0x1046d6 (0x76d7 bytes) 000449[1] 1788620103.265263 trunc from 0x7fe443 to 0x2a9c62 (0x5547e0 bytes) 000450[0] 1788620103.336363 write_OD 0x039000 thru 0x03afff (0x2000 bytes) 000451[0] 1788620103.431560 trunc from 0x2a9c63 to 0x196302 (0x113960 bytes) 000452[1] 1788620103.522076 punch from 0x079d4a to 0x08017c (0x6433 bytes) 000456[1] 1788620103.617851 punch from 0x1180aa to 0x118a01 (0x958 bytes) 000457[0] 1788620103.647092 punch from 0x0f76fb to 0x0f9745 (0x204b bytes) 000458[0] 1788620103.682374 punch from 0x0728d3 to 0x0813eb (0xeb19 bytes) 000459[0] 1788620103.724408 read_OD 0x0c3000 thru 0x0c5fff (0x3000 bytes) 000461[0] 1788620103.756361 trunc from 0x196303 to 0x0d43b8 (0xc1f4a bytes) 000463[1] 1788620103.811670 write 0x290000 thru 0x2944f1 (0x44f2 bytes) 000464[1] 1788620103.910201 mapread 0x220000 thru 0x2252ef (0x52f0 bytes) 000465[0] 1788620104.033670 punch from 0x13dcc2 to 0x1429c3 (0x4d02 bytes) 000469[0] 1788620104.149360 read_OD 0x1b7000 thru 0x1bffff (0x9000 bytes) 000470[1] 1788620104.211758 mapread 0x284000 thru 0x28dcbf (0x9cc0 bytes) 000471[0] 1788620104.282799 mapwrite 0x417000 thru 0x424f91 (0xdf92 bytes) 000472[1] 1788620104.599020 mapwrite 0x0be000 thru 0x0bef6e (0xf6f bytes) 000473[0] 1788620104.800449 read_OD 0x0b0000 thru 0x0b9fff (0xa000 bytes) 000475[0] 1788620104.884167 mapwrite 0x699000 thru 0x69bd23 (0x2d24 bytes) 000477[0] 1788620105.142237 read_OD 0x416000 thru 0x416fff (0x1000 bytes) 000478[1] 1788620105.167412 write 0x03d000 thru 0x03ef81 (0x1f82 bytes) 000480[1] 1788620105.195509 mapwrite 0x5fc000 thru 0x607111 (0xb112 bytes) 000482[0] 1788620105.311235 mapread 0x108000 thru 0x114463 (0xc464 bytes) 000483[0] 1788620105.373974 mapwrite 0x1e2000 thru 0x1e774e (0x574f bytes) 000484[0] 1788620105.529570 read_OD 0x061000 thru 0x06efff (0xe000 bytes) 000485[0] 1788620105.556328 write_OD 0x5e7000 thru 0x5ecfff (0x6000 bytes) 000486[0] 1788620105.632444 read_OD 0x30f000 thru 0x312fff (0x4000 bytes) 000487[1] 1788620105.646524 read 0x0c3000 thru 0x0c3be5 (0xbe6 bytes) 000488[1] 1788620105.653610 read 0x3ac000 thru 0x3baafb (0xeafc bytes) 000489[0] 1788620105.666555 trunc from 0x69bd24 to 0x684559 (0x177ca bytes) 000491[0] 1788620105.741306 write_OD 0x2d8000 thru 0x2e5fff (0xe000 bytes) 000493[1] 1788620105.846947 read 0x582000 thru 0x587d7a (0x5d7b bytes) 000496[1] 1788620105.862761 mapread 0x1a3000 thru 0x1af986 (0xc987 bytes) 000498[1] 1788620105.898537 read 0x4b3000 thru 0x4c0e56 (0xde57 bytes) 000499[0] 1788620105.910432 trunc from 0x68455a to 0x5fda63 (0x86af6 bytes) All operations completed A-OK! PASS 16a (63s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 10:55:11 (1788620111) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.40229 s, 20.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.553586 s, 15.2 MB/s Chance of close/open is 1 in 50 Seed set to 115 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole truncating to largest ever: 0x271fc4 000005[1] 1788620116.260745 trunc from 00000000 to 0x271fc3 (0x271fc4 bytes) truncating to largest ever: 0x653bbf 000007[0] 1788620116.357470 trunc from 0x271fc4 to 0x653bbe (0x3e1bfb bytes) 000008[1] 1788620116.493254 read 0x29735e thru 0x299321 (0x1fc4 bytes) 000009[0] 1788620116.524992 read 0x5d11d0 thru 0x5d1374 (0x1a5 bytes) 000011[1] 1788620116.570250 punch from 0x1237d4 to 0x1251d8 (0x1a05 bytes) 000013[0] 1788620116.690910 trunc from 0x653bbf to 0x550bd4 (0x102fea bytes) 000014[0] 1788620116.751327 write 0x01e359 thru 0x01eb1d (0x7c5 bytes) 000015[1] 1788620116.763605 read 0x1e2aaa thru 0x1e417b (0x16d2 bytes) 000016[0] 1788620116.801932 write 0x0a2637 thru 0x0a2f0c (0x8d6 bytes) 000017[1] 1788620116.889651 write 0x69b6dd thru 0x69d6b5 (0x1fd9 bytes) 000018[1] 1788620116.921646 write 0x4490ba thru 0x44a1a5 (0x10ec bytes) 000019[0] 1788620116.947388 read 0x2c06cf thru 0x2c206a (0x199c bytes) 000021[1] 1788620117.000129 mapread 0x07705a thru 0x07748c (0x433 bytes) 000024[1] 1788620117.058716 write 0x62600d thru 0x626232 (0x226 bytes) 000025[0] 1788620117.083244 write 0x6b9d12 thru 0x6bac33 (0xf22 bytes) 000028[1] 1788620117.195538 mapread 0x53a170 thru 0x53c139 (0x1fca bytes) 000029[0] 1788620117.336777 trunc from 0x6bac34 to 0x171ee4 (0x548d4f bytes) 000030[0] 1788620117.386093 punch from 0x1351af to 0x13554e (0x3a0 bytes) 000031[1] 1788620117.425671 trunc from 0x171ee5 to 0x4f5fc8 (0x3840e4 bytes) 000033[1] 1788620117.642744 punch from 0x491323 to 0x493202 (0x1ee0 bytes) 000035[1] 1788620117.659287 read 0x2858f7 thru 0x286a46 (0x1150 bytes) 000036[0] 1788620117.703703 mapread 0x1adcd8 thru 0x1ae608 (0x931 bytes) 000037[0] 1788620117.900143 write 0x987c36 thru 0x9898e1 (0x1cac bytes) 000039[0] 1788620118.036184 punch from 0x6009bf to 0x6028a0 (0x1ee2 bytes) 000040[0] 1788620118.098500 mapwrite 0x26a179 thru 0x26aa02 (0x88a bytes) 000042[1] 1788620118.304331 read 0x31d6d5 thru 0x31dcbe (0x5ea bytes) 000047[1] 1788620118.469229 write 0x4ecdb4 thru 0x4ed3e8 (0x635 bytes) 000051[0] 1788620118.616177 mapwrite 0x05675f thru 0x057225 (0xac7 bytes) 000053[0] 1788620118.929015 trunc from 0x9898e2 to 0x03e167 (0x94b77a bytes) truncating to largest ever: 0x992f60 000054[0] 1788620119.019669 trunc from 0x03e168 to 0x992f5f (0x954df8 bytes) 000055[0] 1788620119.049562 trunc from 0x992f60 to 0x46fe03 (0x52315c bytes) 000056[1] 1788620119.141573 write 0x046c38 thru 0x047c77 (0x1040 bytes) 000058[0] 1788620119.284752 mapread 0x428804 thru 0x42a189 (0x1986 bytes) 000059[1] 1788620119.373902 write 0x7269ae thru 0x728966 (0x1fb9 bytes) 000061[1] 1788620119.472766 read 0x716bd5 thru 0x71810d (0x1539 bytes) 000062[0] 1788620119.478575 trunc from 0x728967 to 0x60308c (0x1258da bytes) 000063[0] 1788620119.579551 read 0x2abb1a thru 0x2ac914 (0xdfb bytes) 000064[1] 1788620119.590755 mapwrite 0x286fd0 thru 0x2877d5 (0x806 bytes) 000065[1] 1788620119.787349 mapread 0x3044e8 thru 0x305644 (0x115d bytes) 000066[0] 1788620119.877085 mapread 0x200595 thru 0x201664 (0x10d0 bytes) 000068[0] 1788620120.113857 punch from 0x1db35a to 0x1db9fa (0x6a1 bytes) 000069[0] 1788620120.223593 trunc from 0x60308d to 0x977d68 (0x374cdc bytes) 000071[0] 1788620120.374551 mapread 0x722c1d thru 0x7231be (0x5a2 bytes) 000072[0] 1788620120.436916 punch from 0x113da1 to 0x114740 (0x9a0 bytes) 000073[1] 1788620120.512546 write 0x2a3ad8 thru 0x2a5353 (0x187c bytes) 000075[0] 1788620120.615974 trunc from 0x977d69 to 0x1eac02 (0x78d166 bytes) 000076[0] 1788620120.689143 write 0x191a55 thru 0x191b05 (0xb1 bytes) 000077[1] 1788620120.804347 write 0x1c2f97 thru 0x1c3e6d (0xed7 bytes) 000090[0] 1788620121.383367 trunc from 0x1eac03 to 0x992645 (0x7a7a43 bytes) 000094[0] 1788620121.811632 trunc from 0x992646 to 0x5e145b (0x3b11ea bytes) 000095[0] 1788620121.917025 punch from 0x1ba6a7 to 0x1bb490 (0xdea bytes) 000096[1] 1788620121.953615 mapwrite 0x8baaab thru 0x8bbec1 (0x1417 bytes) 000100[0] 1788620122.302110 punch from 0x6368f5 to 0x636992 (0x9e bytes) 000102[0] 1788620122.419975 trunc from 0x8bbec2 to 0x7a4263 (0x117c5e bytes) 000106[0] 1788620122.561514 trunc from 0x7a4264 to 0x59d87e (0x2069e5 bytes) 000107[0] 1788620122.632554 punch from 0x0dca9f to 0x0de2cf (0x1831 bytes) 000108[0] 1788620122.647504 read 0x37fc96 thru 0x380275 (0x5e0 bytes) 000112[0] 1788620122.702205 punch from 0x57c07e to 0x57d673 (0x15f6 bytes) 000113[0] 1788620122.730122 mapread 0x36766d thru 0x369206 (0x1b9a bytes) 000115[0] 1788620122.859556 mapwrite 0x220d6e thru 0x22249a (0x172d bytes) 000116[1] 1788620123.025203 punch from 0x082b97 to 0x0835be (0xa28 bytes) 000122[0] 1788620123.307986 read 0x50737d thru 0x50802f (0xcb3 bytes) 000123[1] 1788620123.360191 punch from 0x029345 to 0x029f13 (0xbcf bytes) 000124[0] 1788620123.400532 mapwrite 0x174430 thru 0x17479e (0x36f bytes) 000126[1] 1788620123.577222 write 0x0f0b02 thru 0x0f1d73 (0x1272 bytes) 000128[1] 1788620123.675910 write 0x068f0e thru 0x0690bb (0x1ae bytes) 000129[0] 1788620123.757394 write 0x1f6dad thru 0x1f7281 (0x4d5 bytes) 000130[1] 1788620123.911543 mapwrite 0x18d15c thru 0x18db02 (0x9a7 bytes) 000136[0] 1788620124.439627 mapwrite 0x2673fa thru 0x269374 (0x1f7b bytes) 000137[0] 1788620124.774755 punch from 0x3d810c to 0x3d9b77 (0x1a6c bytes) 000138[1] 1788620124.909300 read 0x5874a4 thru 0x5893be (0x1f1b bytes) 000140[0] 1788620125.101457 trunc from 0x59d87f to 0x66bade (0xce260 bytes) 000141[0] 1788620125.195723 write 0x39b670 thru 0x39bffd (0x98e bytes) 000142[0] 1788620125.209958 mapread 0x49e58d thru 0x49ff36 (0x19aa bytes) 000145[1] 1788620125.258096 punch from 0x0d19d3 to 0x0d2d4c (0x137a bytes) 000146[0] 1788620125.299317 trunc from 0x66badf to 0x6af1bc (0x436de bytes) 000147[1] 1788620125.342551 mapread 0x213739 thru 0x213d7f (0x647 bytes) 000148[0] 1788620125.441679 read 0x045927 thru 0x046f4a (0x1624 bytes) 000149[0] 1788620125.470555 trunc from 0x6af1bd to 0x06f977 (0x63f845 bytes) 000150[0] 1788620125.556263 trunc from 0x06f978 to 0x5fb3ae (0x58ba37 bytes) 000151[1] 1788620125.616191 read 0x117dbd thru 0x1189c1 (0xc05 bytes) 000154[0] 1788620125.667638 write 0x871746 thru 0x871e88 (0x743 bytes) 000159[0] 1788620125.718758 mapread 0x1e7081 thru 0x1e7b3f (0xabf bytes) 000161[1] 1788620125.787190 mapread 0x1ef668 thru 0x1f128f (0x1c28 bytes) 000163[1] 1788620125.840108 read 0x4682d7 thru 0x4687d4 (0x4fe bytes) 000164[1] 1788620125.868516 trunc from 0x871e89 to 0x8c920a (0x57382 bytes) 000165[0] 1788620125.935506 mapread 0x794207 thru 0x795d19 (0x1b13 bytes) 000167[1] 1788620126.007099 write 0x21d94e thru 0x21ea0e (0x10c1 bytes) 000170[0] 1788620126.060592 write 0x3f03dd thru 0x3f0a53 (0x677 bytes) 000171[0] 1788620126.097432 punch from 0x7bd1ab to 0x7bdfa3 (0xdf9 bytes) 000172[0] 1788620126.122646 mapwrite 0x65b465 thru 0x65c865 (0x1401 bytes) 000175[1] 1788620126.171540 mapwrite 0x166f6a thru 0x167c63 (0xcfa bytes) 000177[0] 1788620126.291187 mapwrite 0x8e8796 thru 0x8e9f08 (0x1773 bytes) 000178[0] 1788620126.390250 punch from 0x3c056f to 0x3c1a89 (0x151b bytes) 000182[1] 1788620126.469633 punch from 0x03a19a to 0x03bcfc (0x1b63 bytes) 000183[0] 1788620126.508224 mapwrite 0x753139 thru 0x753483 (0x34b bytes) 000185[0] 1788620126.595323 trunc from 0x8e9f09 to 0x98022d (0x96325 bytes) 000186[1] 1788620126.656207 mapread 0x75bb79 thru 0x75ce5d (0x12e5 bytes) 000188[0] 1788620126.734536 mapwrite 0x64977b thru 0x649f33 (0x7b9 bytes) 000190[1] 1788620126.844711 trunc from 0x98022e to 0x5635ce (0x41cc5f bytes) 000193[0] 1788620126.953276 mapread 0x316898 thru 0x318233 (0x199c bytes) 000195[0] 1788620127.020166 read 0x1e3649 thru 0x1e43d0 (0xd88 bytes) 000196[1] 1788620127.286641 punch from 0x015b5d to 0x016043 (0x4e7 bytes) 000197[0] 1788620127.343594 mapwrite 0x81f364 thru 0x81f96e (0x60b bytes) 000198[0] 1788620127.664130 punch from 0x7f2f2e to 0x7f4a52 (0x1b25 bytes) 000200[0] 1788620127.724973 read 0x6e1949 thru 0x6e1ff6 (0x6ae bytes) 000201[1] 1788620127.745544 read 0x719c19 thru 0x71a18b (0x573 bytes) 000202[0] 1788620127.772760 mapwrite 0x307a9a thru 0x308b5e (0x10c5 bytes) 000204[1] 1788620127.847621 read 0x638eeb thru 0x63aa37 (0x1b4d bytes) 000205[0] 1788620127.875524 mapwrite 0x8f3574 thru 0x8f412e (0xbbb bytes) 000206[1] 1788620128.067540 mapwrite 0x9ed426 thru 0x9ee34b (0xf26 bytes) 000208[0] 1788620128.429899 mapwrite 0x3d25a5 thru 0x3d3210 (0xc6c bytes) 000209[1] 1788620128.730247 write 0x9de6e1 thru 0x9df1a7 (0xac7 bytes) 000210[1] 1788620128.784702 punch from 0x233793 to 0x235274 (0x1ae2 bytes) 000212[0] 1788620128.794648 trunc from 0x9ee34c to 0x294f53 (0x7593f8 bytes) 000213[1] 1788620128.938132 write 0x7ad61d thru 0x7ae90d (0x12f1 bytes) 000215[0] 1788620129.002936 mapwrite 0x90ef8a thru 0x9103b4 (0x142b bytes) truncating to largest ever: 0x9d3ffa 000216[1] 1788620129.150377 trunc from 0x9103b5 to 0x9d3ff9 (0xc3c45 bytes) 000218[0] 1788620129.205695 mapread 0x7362d8 thru 0x7370d3 (0xdfc bytes) 000219[0] 1788620129.242634 mapread 0x1e3188 thru 0x1e4fe2 (0x1e5b bytes) 000220[0] 1788620129.278713 write 0x47779d thru 0x4786d2 (0xf36 bytes) 000221[1] 1788620129.318219 punch from 0x7b56ed to 0x7b767f (0x1f93 bytes) 000222[1] 1788620129.354493 trunc from 0x9d3ffa to 0x6fabf0 (0x2d9409 bytes) 000223[0] 1788620129.416649 punch from 0x2074ed to 0x207802 (0x316 bytes) 000225[1] 1788620129.491255 trunc from 0x6fabf1 to 0x713a0a (0x18e1a bytes) 000226[0] 1788620129.557333 write 0x5b72c6 thru 0x5b8eb2 (0x1bed bytes) 000229[1] 1788620129.656481 punch from 0x040ff8 to 0x041142 (0x14b bytes) 000230[0] 1788620129.677641 mapread 0x1d3c85 thru 0x1d43e2 (0x75e bytes) 000231[1] 1788620129.715920 mapwrite 0x55fd8d thru 0x560f4a (0x11be bytes) 000232[0] 1788620129.836806 mapread 0x25e13a thru 0x25ef9b (0xe62 bytes) 000233[0] 1788620129.856701 write 0x248d8f thru 0x24a7ae (0x1a20 bytes) 000234[1] 1788620129.891935 trunc from 0x713a0b to 0x14f1d2 (0x5c4838 bytes) 000239[0] 1788620130.092163 write 0x486ede thru 0x4889f6 (0x1b19 bytes) 000240[0] 1788620130.168026 mapwrite 0x24f694 thru 0x24ff6a (0x8d7 bytes) 000241[0] 1788620130.358090 mapwrite 0x90a043 thru 0x90b1dc (0x119a bytes) 000242[1] 1788620130.607852 write 0x18a819 thru 0x18bbaa (0x1392 bytes) 000243[1] 1788620130.728945 mapwrite 0x19dfe5 thru 0x19eb57 (0xb73 bytes) 000244[0] 1788620131.031179 write 0x738918 thru 0x73a691 (0x1d7a bytes) 000245[1] 1788620131.131767 punch from 0x0971e8 to 0x0975dd (0x3f6 bytes) 000247[1] 1788620131.200499 punch from 0x2b92e9 to 0x2bb276 (0x1f8e bytes) 000251[0] 1788620131.278156 trunc from 0x90b1dd to 0x9863b6 (0x7b1da bytes) 000253[1] 1788620131.353180 mapwrite 0x7d9ce0 thru 0x7db8fd (0x1c1e bytes) 000256[1] 1788620131.602214 punch from 0x01c62a to 0x01dea4 (0x187b bytes) 000259[0] 1788620131.690073 mapread 0x8a7330 thru 0x8a8555 (0x1226 bytes) 000260[1] 1788620131.790991 read 0x8d9ef3 thru 0x8dbc95 (0x1da3 bytes) 000261[1] 1788620131.833323 punch from 0x3c6284 to 0x3c826b (0x1fe8 bytes) 000262[0] 1788620131.863400 mapwrite 0x3c25d6 thru 0x3c2ff3 (0xa1e bytes) 000264[0] 1788620132.103655 trunc from 0x9863b7 to 0x84f7d7 (0x136bdf bytes) 000267[1] 1788620132.154615 read 0x57e905 thru 0x57ff82 (0x167e bytes) 000268[1] 1788620132.189129 punch from 0x6663ec to 0x666d10 (0x925 bytes) 000269[1] 1788620132.276192 mapwrite 0x85cf01 thru 0x85cf2d (0x2d bytes) 000274[1] 1788620132.502316 mapwrite 0x328d63 thru 0x3298a8 (0xb46 bytes) 000275[0] 1788620132.592489 mapwrite 0x4e23c9 thru 0x4e33d8 (0x1010 bytes) 000277[0] 1788620132.834909 read 0x1fd03f thru 0x1fdfec (0xfae bytes) 000278[0] 1788620132.886224 trunc from 0x85cf2e to 0x3a4e35 (0x4b80f8 bytes) 000280[1] 1788620133.079652 read 0x0a0427 thru 0x0a1728 (0x1302 bytes) 000282[1] 1788620133.273344 mapread 0x22d64c thru 0x22e3b4 (0xd69 bytes) 000285[0] 1788620133.285592 write 0x2fc5bc thru 0x2fdbcc (0x1611 bytes) 000286[0] 1788620133.365940 write 0x770d0a thru 0x771ee1 (0x11d8 bytes) 000288[1] 1788620133.415432 read 0x26cef5 thru 0x26d7ab (0x8b7 bytes) 000290[1] 1788620133.471870 trunc from 0x771ee2 to 0x4b1070 (0x2c0e71 bytes) 000291[1] 1788620133.560600 read 0x15fae5 thru 0x15fbb6 (0xd2 bytes) 000293[0] 1788620133.640000 punch from 0x2a7c92 to 0x2a9b78 (0x1ee7 bytes) 000294[0] 1788620133.696274 trunc from 0x4b1071 to 0x6c15d6 (0x210566 bytes) 000297[1] 1788620133.781639 mapread 0x452421 thru 0x453128 (0xd08 bytes) 000300[0] 1788620133.953291 mapread 0x694f29 thru 0x69653e (0x1616 bytes) 000301[1] 1788620134.055408 write 0x0081a2 thru 0x00848c (0x2eb bytes) 000302[1] 1788620134.170574 mapwrite 0x6578ec thru 0x659542 (0x1c57 bytes) 000308[0] 1788620134.876370 read 0x5b94c7 thru 0x5baf8e (0x1ac8 bytes) 000309[1] 1788620134.943584 read 0x2ce9f1 thru 0x2ced20 (0x330 bytes) 000311[0] 1788620134.994456 write 0x02ed47 thru 0x02fafc (0xdb6 bytes) 000313[0] 1788620135.103932 mapwrite 0x13fea4 thru 0x140d72 (0xecf bytes) 000314[0] 1788620135.240063 punch from 0x429acb to 0x42a602 (0xb38 bytes) 000316[0] 1788620135.275729 punch from 0x339433 to 0x33958d (0x15b bytes) 000318[1] 1788620135.325575 read 0x1cb5b7 thru 0x1cd199 (0x1be3 bytes) 000319[1] 1788620135.376365 punch from 0x3b39a2 to 0x3b4672 (0xcd1 bytes) 000320[0] 1788620135.438176 trunc from 0x6c15d7 to 0x350650 (0x370f86 bytes) 000324[0] 1788620135.542180 read 0x011150 thru 0x0117fa (0x6ab bytes) 000325[1] 1788620135.676260 trunc from 0x350651 to 0x581faf (0x23195f bytes) 000329[1] 1788620135.902487 read 0x2fdd61 thru 0x2fe731 (0x9d1 bytes) 000330[1] 1788620135.978855 read 0x42c817 thru 0x42ca7d (0x267 bytes) 000331[1] 1788620136.031187 read 0x428594 thru 0x428b24 (0x591 bytes) 000332[0] 1788620136.038781 trunc from 0x581fb0 to 0x5e159c (0x5f5ed bytes) 000335[0] 1788620136.249551 mapread 0x29535f thru 0x296931 (0x15d3 bytes) 000336[1] 1788620136.340127 trunc from 0x5e159d to 0x959467 (0x377ecb bytes) 000337[0] 1788620136.409135 read 0x478c40 thru 0x47a4db (0x189c bytes) 000338[1] 1788620136.452757 trunc from 0x959468 to 0x2fb043 (0x65e424 bytes) 000340[0] 1788620136.559753 write 0x2e943a thru 0x2e956b (0x132 bytes) 000342[1] 1788620136.661655 mapwrite 0x1a1096 thru 0x1a1473 (0x3de bytes) 000344[1] 1788620136.977023 mapread 0x03036d thru 0x031b43 (0x17d7 bytes) 000346[0] 1788620137.106998 write 0x911d4d thru 0x9135e9 (0x189d bytes) 000347[0] 1788620137.168116 read 0x45c58f thru 0x45dbad (0x161f bytes) 000348[0] 1788620137.227759 mapwrite 0x8e0a71 thru 0x8e1f32 (0x14c2 bytes) 000349[0] 1788620137.359207 punch from 0x212593 to 0x213c7c (0x16ea bytes) 000350[1] 1788620137.403934 mapwrite 0x28a099 thru 0x28a2d5 (0x23d bytes) 000351[0] 1788620137.595028 punch from 0x7ef322 to 0x7f10db (0x1dba bytes) 000352[1] 1788620137.666008 mapwrite 0x8cbcb9 thru 0x8cbe78 (0x1c0 bytes) 000353[1] 1788620137.805445 mapwrite 0x355990 thru 0x35643d (0xaae bytes) 000354[0] 1788620138.050041 mapread 0x4489f6 thru 0x449b7c (0x1187 bytes) 000356[1] 1788620138.177469 read 0x5c056b thru 0x5c1cfa (0x1790 bytes) 000358[0] 1788620138.277847 trunc from 0x9135ea to 0x742c7d (0x1d096c bytes) 000359[0] 1788620138.414482 punch from 0x401d1c to 0x4031ba (0x149f bytes) 000360[0] 1788620138.429567 write 0x27010c thru 0x2718aa (0x179f bytes) 000361[1] 1788620138.491477 mapread 0x05dcb6 thru 0x05e4de (0x829 bytes) 000362[0] 1788620138.633343 write 0x895602 thru 0x897057 (0x1a56 bytes) 000363[1] 1788620138.733284 read 0x053e33 thru 0x054d66 (0xf34 bytes) 000364[1] 1788620138.864963 mapwrite 0x18e399 thru 0x18f114 (0xd7c bytes) 000365[0] 1788620139.094475 mapread 0x1d5c31 thru 0x1d5d70 (0x140 bytes) 000367[0] 1788620139.244262 mapwrite 0x78dfc3 thru 0x78fe44 (0x1e82 bytes) 000368[0] 1788620139.326470 trunc from 0x897058 to 0x66afe7 (0x22c070 bytes) 000369[0] 1788620139.408493 mapread 0x4e9813 thru 0x4e9c36 (0x424 bytes) 000370[0] 1788620139.468410 mapread 0x433b54 thru 0x4346cd (0xb7a bytes) 000371[1] 1788620139.560350 mapwrite 0x04ed59 thru 0x04fc0c (0xeb4 bytes) 000372[0] 1788620139.724792 read 0x2c5080 thru 0x2c6a40 (0x19c1 bytes) 000373[1] 1788620139.786536 write 0x3cb241 thru 0x3cb87f (0x63f bytes) 000374[0] 1788620139.835307 punch from 0x3849d9 to 0x3860d4 (0x16fc bytes) 000377[0] 1788620139.867695 mapread 0x14305b thru 0x14391d (0x8c3 bytes) 000378[0] 1788620139.926923 read 0x1899b9 thru 0x189c43 (0x28b bytes) 000380[1] 1788620139.985346 mapwrite 0x2c0b26 thru 0x2c2771 (0x1c4c bytes) 000381[1] 1788620140.149175 punch from 0x5e3a14 to 0x5e4a4a (0x1037 bytes) 000383[0] 1788620140.193669 mapwrite 0x55909b thru 0x559776 (0x6dc bytes) 000384[1] 1788620140.313641 write 0x15b1d2 thru 0x15cc4c (0x1a7b bytes) 000386[0] 1788620140.366622 punch from 0x489801 to 0x48a407 (0xc07 bytes) 000387[1] 1788620140.388627 punch from 0x4173b1 to 0x418669 (0x12b9 bytes) 000388[1] 1788620140.471161 write 0x179821 thru 0x17a91c (0x10fc bytes) 000389[1] 1788620140.535138 trunc from 0x66afe8 to 0x6c51bd (0x5a1d6 bytes) 000390[1] 1788620140.587504 mapread 0x6361f1 thru 0x636590 (0x3a0 bytes) 000391[0] 1788620140.624412 trunc from 0x6c51be to 0x7b2f12 (0xedd55 bytes) 000392[0] 1788620140.726847 punch from 0x141175 to 0x142b9f (0x1a2b bytes) 000393[0] 1788620140.741076 write 0x2659fe thru 0x26791e (0x1f21 bytes) 000394[1] 1788620140.790942 mapread 0x65688a thru 0x656987 (0xfe bytes) 000401[1] 1788620140.923776 read 0x0eb876 thru 0x0ec9b8 (0x1143 bytes) 000402[0] 1788620140.946255 punch from 0x2dcfe0 to 0x2de505 (0x1526 bytes) 000406[1] 1788620141.015411 write 0x2de1b4 thru 0x2deff4 (0xe41 bytes) 000408[0] 1788620141.051524 write 0x4b7729 thru 0x4b958a (0x1e62 bytes) 000409[0] 1788620141.097424 mapwrite 0x643434 thru 0x64498d (0x155a bytes) 000412[0] 1788620141.202487 trunc from 0x7b2f13 to 0x923a44 (0x170b32 bytes) 000413[1] 1788620141.246370 write 0x403e75 thru 0x404e4c (0xfd8 bytes) 000414[0] 1788620141.286609 trunc from 0x923a45 to 0x4af1f0 (0x474854 bytes) 000415[0] 1788620141.326776 write 0x0e751b thru 0x0e9200 (0x1ce6 bytes) 000416[0] 1788620141.337440 mapwrite 0x660b6d thru 0x661126 (0x5ba bytes) 000421[1] 1788620141.441431 write 0x2fd4d2 thru 0x2fdd8a (0x8b9 bytes) 000422[0] 1788620141.499500 punch from 0x0d8e8e to 0x0d98ec (0xa5f bytes) 000423[0] 1788620141.547279 mapwrite 0x302d6d thru 0x304d38 (0x1fcc bytes) 000424[1] 1788620141.611516 punch from 0x22cd11 to 0x22dc85 (0xf75 bytes) 000425[1] 1788620141.634255 mapwrite 0x709e41 thru 0x70b66f (0x182f bytes) 000427[1] 1788620141.740457 mapread 0x5d386b thru 0x5d54c1 (0x1c57 bytes) 000428[0] 1788620141.759151 mapread 0x454fa8 thru 0x455b23 (0xb7c bytes) 000429[0] 1788620141.809054 read 0x32cde2 thru 0x32daab (0xcca bytes) 000430[0] 1788620141.834776 punch from 0x68f19c to 0x68f5b5 (0x41a bytes) 000433[0] 1788620141.959918 trunc from 0x70b670 to 0x7d411e (0xc8aaf bytes) 000434[0] 1788620142.002021 trunc from 0x7d411f to 0x64c53c (0x187be2 bytes) 000435[1] 1788620142.017247 mapread 0x5a346f thru 0x5a35a6 (0x138 bytes) 000436[0] 1788620142.060169 write 0x6b351f thru 0x6b4379 (0xe5b bytes) 000437[1] 1788620142.105145 punch from 0x369a15 to 0x36a196 (0x782 bytes) 000438[1] 1788620142.141747 write 0x4a232b thru 0x4a36e9 (0x13bf bytes) 000439[0] 1788620142.167837 read 0x2e7d26 thru 0x2e86e5 (0x9c0 bytes) 000440[0] 1788620142.190716 trunc from 0x6b437a to 0x79290e (0xde595 bytes) 000441[1] 1788620142.259349 trunc from 0x79290f to 0x6ed18f (0xa577f bytes) 000444[0] 1788620142.306156 mapread 0x3f8b77 thru 0x3f8d0f (0x199 bytes) 000445[1] 1788620142.362201 write 0x611f9f thru 0x61337d (0x13df bytes) 000446[1] 1788620142.388917 trunc from 0x6ed190 to 0x6a55ce (0x47bc1 bytes) 000449[1] 1788620142.506423 write 0x6aa019 thru 0x6aa048 (0x30 bytes) 000450[0] 1788620142.512235 write 0x40c63c thru 0x40d8c8 (0x128d bytes) 000452[1] 1788620142.570844 mapread 0x29b767 thru 0x29c814 (0x10ae bytes) 000453[1] 1788620142.603629 mapwrite 0x7abea5 thru 0x7acd0c (0xe68 bytes) 000454[0] 1788620142.690116 punch from 0x761dbe to 0x763bae (0x1df1 bytes) 000455[0] 1788620142.730152 read 0x31fb4f thru 0x31fc5b (0x10d bytes) 000456[1] 1788620142.768132 write 0x668d84 thru 0x66ac85 (0x1f02 bytes) 000457[1] 1788620142.810290 read 0x369e8a thru 0x36bcf8 (0x1e6f bytes) 000458[1] 1788620142.818100 punch from 0x60a29e to 0x60b351 (0x10b4 bytes) 000459[1] 1788620142.840130 trunc from 0x7acd0d to 0x0719c7 (0x73b345 bytes) 000462[0] 1788620142.942503 mapread 0x0564d9 thru 0x0569ad (0x4d5 bytes) 000464[1] 1788620143.005919 read 0x033145 thru 0x0334f6 (0x3b2 bytes) 000465[1] 1788620143.041541 write 0x99984e thru 0x99b18e (0x1941 bytes) 000469[0] 1788620143.084907 write 0x10330e thru 0x1047aa (0x149d bytes) 000472[0] 1788620143.163958 mapread 0x8485b4 thru 0x8496a8 (0x10f5 bytes) 000475[1] 1788620143.233260 punch from 0x5c5f1f to 0x5c68c9 (0x9ab bytes) 000476[1] 1788620143.252333 trunc from 0x99b18f to 0x732144 (0x26904a bytes) 000477[0] 1788620143.290769 punch from 0x3eecc5 to 0x3efb15 (0xe51 bytes) 000478[1] 1788620143.324877 write 0x7e04de thru 0x7e230a (0x1e2d bytes) 000479[0] 1788620143.343445 punch from 0x5a2f95 to 0x5a3c85 (0xcf1 bytes) 000481[1] 1788620143.388472 mapwrite 0x03fe58 thru 0x040305 (0x4ae bytes) 000485[1] 1788620143.666880 punch from 0x2eb51b to 0x2ebe92 (0x978 bytes) 000486[1] 1788620143.692549 write 0x8403c9 thru 0x840a54 (0x68c bytes) 000487[0] 1788620143.720982 punch from 0x1c7dbd to 0x1c888e (0xad2 bytes) 000489[1] 1788620143.812225 punch from 0x1787d0 to 0x179673 (0xea4 bytes) 000490[1] 1788620143.854266 trunc from 0x840a55 to 0x03c06c (0x8049e8 bytes) 000494[1] 1788620143.949666 trunc from 0x03c06d to 0x61e345 (0x5e22d9 bytes) 000495[0] 1788620144.089524 punch from 0x0dacdd to 0x0db4aa (0x7ce bytes) 000496[1] 1788620144.193352 mapread 0x30678a thru 0x3079e4 (0x125b bytes) 000497[1] 1788620144.311365 mapwrite 0x5894fd thru 0x58a79d (0x12a1 bytes) 000498[1] 1788620144.527619 mapwrite 0x74d5ee thru 0x74e8b5 (0x12c8 bytes) 000499[0] 1788620144.686446 trunc from 0x74e8b6 to 0x0987bf (0x6b60f6 bytes) 000500[0] 1788620144.792550 read 0x060c5c thru 0x061bc3 (0xf68 bytes) All operations completed A-OK! PASS 16b (39s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 10:55:51 (1788620151) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 10:55:55 (1788620155) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.422499 s, 19.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.433438 s, 19.4 MB/s ldlm.namespaces.MGC192.168.204.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9af283332800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9af285452800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9af285452800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9af285452800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.345 s, 7.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.86913 s, 27.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.96746 s, 35.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53665 s, 16.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.38921 s, 12.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.95605 s, 26.5 MB/s PASS 16d (54s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 10:56:49 (1788620209) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342456 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119156 s, 8.8 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 10:56:57 (1788620217) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 10:57:26 (1788620246) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 10:57:55 (1788620275) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 10:58:04 (1788620284) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (13s) == sanityn test 16j: race dio with buffered i/o ========== 10:58:18 (1788620298) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.42706 s, 10.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.65889 s, 15.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.84136 s, 6.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.55735 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.59993 s, 7.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35504 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.88237 s, 5.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.81595 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.86158 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.36442 s, 6.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.43712 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.39095 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.59536 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.03279 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.23876 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.49963 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.0713 s, 5.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.51744 s, 4.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.69099 s, 4.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (56s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 10:59:14 (1788620354) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Sat Sep 5 11:00:09 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Sat Sep 5 11:00:09 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Sat Sep 5 11:00:10 EDT 2026 + fsx 4 done: rc=0 Sat Sep 5 11:00:10 EDT 2026 + fsx 5 done: rc=0 Sat Sep 5 11:00:10 EDT 2026 PASS 16k (67s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 11:00:21 (1788620421) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 11:00:34 (1788620434) mmap test1: basic mmap operation (PASS, 0.12405s) mmap test2: MAP_PRIVATE not write back (PASS, 0.1066s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.0942s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6286s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.17319s) mmap test6: check mmap write/read content on two nodes (PASS, 0.31893s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.8329s) mmap test11: multi-thread shared mmap (PASS, 1.3613s) mmap test10: mtime not change for readonly mmap access (PASS, 1.2504s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 11:01:03 (1788620463) oleg455-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 11:01:09 (1788620469) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 11:01:16 (1788620476) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 11:01:25 (1788620485) now is 1788620488 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 new atime is 1788620549 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 11:02:36 (1788620556) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 11:02:42 (1788620562) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 11:02:50 (1788620570) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 11:02:59 (1788620579) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 11:03:03 (1788620583) utime: good mknod times 1788620584 <= 1788620585 <= 1788620585 for /mnt/lustre/f26a utime: good mknod times 1788620584 <= 1788620585 <= 1788620585 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 11:03:12 (1788620592) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 11:03:22 (1788620602) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 11:03:24.000000000 -0400 Birth: 2026-09-05 11:03:25.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 11:03:24.000000000 -0400 Birth: 2026-09-05 11:03:25.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 11:03:33 (1788620613) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 11:03:44 (1788620624) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 11:03:54 (1788620634) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg455-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.155@tcp:/lustre 7542784 41984 7495680 1% /mnt/lustre2 PASS 31b (30s) == sanityn test 31r: open-rename(replace) race =========== 11:04:24 (1788620664) -rw-r--r-- 1 root root 0 Sep 5 11:04 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 11:04:36 (1788620676) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43226 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 11:04:44 (1788620684) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 11:04:54 (1788620694) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 11:04:59 (1788620699) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.05263 s, 25.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2638 s, 41.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.0445 s, 17.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.70923 s, 14.1 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.78976 s, 18.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.01666 s, 17.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.96221 s, 26.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9539 s, 26.8 MB/s PASS 32c (38s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 11:05:40 (1788620740) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 11:05:44 (1788620744) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 11:05:47 (1788620747) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 11:05:51 (1788620751) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 11:05:55 (1788620755) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af283332800.ost_server_uuid in IDLE state after 0 sec PASS 34 (157s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 11:08:32 (1788620912) fail_loc=0x80000317 Race attempt 0 Wait for 50539 50565 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50539 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.155@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.204.155@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 - open/close 548 (time 1788620930.05 total 10.00 last 54.80) - open/close 1192 (time 1788620940.07 total 20.02 last 64.29) - open/close 1722 (time 1788620950.08 total 30.03 last 52.93) - open/close 2256 (time 1788620960.08 total 40.04 last 53.37) - open/close 2861 (time 1788620970.09 total 50.04 last 60.47) - open/close 3434 (time 1788620980.10 total 60.05 last 57.23) - open/close 3841 (time 1788620990.11 total 70.07 last 40.65) total: 4000 open/close in 72.87 seconds: 54.89 ops/second PASS 35 (80s) == sanityn test 36: handle ESTALE/open-unlink correctly == 11:09:53 (1788620993) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg455-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21426 s, 43.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.879139 s, 59.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.543971 s, 96.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7418880) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.750899 s, 69.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.797268 s, 65.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7484416) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.536011 s, 97.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7486464) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.753672 s, 69.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7484416) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.974149 s, 53.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7484416) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.815212 s, 64.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.9107 s, 57.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7489536) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05848 s, 49.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7489536) after_dd(7438336) after(7496704) PASS 36 (342s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 11:15:35 (1788621335) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 - create 1076 (time 1788621349.44 total 10.01 last 107.53) - create 2158 (time 1788621359.44 total 20.01 last 108.17) - create 3431 (time 1788621369.45 total 30.01 last 127.26) - create 4588 (time 1788621379.45 total 40.02 last 115.61) - create 5716 (time 1788621389.46 total 50.03 last 112.75) - create 6910 (time 1788621399.47 total 60.03 last 119.32) - create 8102 (time 1788621409.47 total 70.04 last 119.13) - create 9306 (time 1788621419.47 total 80.04 last 120.37) total: 10000 create in 85.97 seconds: 116.32 ops/second PASS 37 (98s) == sanityn test 39a: file mtime does not change after rename ========================================================== 11:17:13 (1788621433) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 11:17:21 (1788621441) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 11:17:31 (1788621451) repeat after cancel_lru_locks PASS 39c (10s) == sanityn test 39d: sync write should update mtime ====== 11:17:41 (1788621461) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 11:17:50 (1788621470) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 11:18:08 (1788621488) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 11:18:25 (1788621505) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 11:18:41 (1788621521) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 11:18:58 (1788621538) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (17s) == sanityn test 41b: pdirops: create vs create ======================================================================== 11:19:15 (1788621555) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 11:19:29 (1788621569) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 11:19:42 (1788621582) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 11:19:54 (1788621594) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 11:20:07 (1788621607) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 11:20:21 (1788621621) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 11:20:34 (1788621634) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 11:20:50 (1788621650) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1150s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 11:40:01 (1788622801) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 11:40:17 (1788622817) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 11:40:32 (1788622832) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 11:40:47 (1788622847) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 11:41:01 (1788622861) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 11:41:16 (1788622876) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 11:41:31 (1788622891) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 11:41:45 (1788622905) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 11:42:00 (1788622920) PASS 43a (73s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 11:43:13 (1788622993) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 11:43:28 (1788623008) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 11:43:42 (1788623022) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 11:43:57 (1788623037) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 11:44:12 (1788623052) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 11:44:26 (1788623066) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 11:44:41 (1788623081) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 11:44:55 (1788623095) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 11:44:59 (1788623099) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (134s) == sanityn test 43k: unlink vs create ==================== 11:47:13 (1788623233) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1355s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 12:09:49 (1788624589) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sat Sep 5 12:09:59 EDT 2026 PASS 44a (16s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 12:10:06 (1788624606) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 12:10:20 (1788624620) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (17s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 12:10:38 (1788624638) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 12:10:51 (1788624651) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 12:11:07 (1788624667) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 12:11:23 (1788624683) Conflict PASS 44g (17s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 12:11:41 (1788624701) Conflict PASS 44h (17s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 12:11:57 (1788624717) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 12:12:01 (1788624721) PASS 45a (125s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 12:14:06 (1788624846) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 12:14:21 (1788624861) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 12:14:34 (1788624874) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 12:14:49 (1788624889) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 12:15:04 (1788624904) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 12:15:17 (1788624917) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 12:15:31 (1788624931) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 12:15:43 (1788624943) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 12:15:46 (1788624946) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1145s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 12:34:51 (1788626091) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 12:35:03 (1788626103) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 12:35:16 (1788626116) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 12:35:30 (1788626130) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 12:35:45 (1788626145) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 12:35:59 (1788626159) Conflict PASS 46f (11s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 12:36:11 (1788626171) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 12:36:23 (1788626183) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 12:36:38 (1788626198) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 12:36:41 (1788626201) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 12:36:45 (1788626205) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 12:36:49 (1788626209) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 12:36:53 (1788626213) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 12:36:57 (1788626217) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 12:37:01 (1788626221) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 12:37:04 (1788626224) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 12:37:08 (1788626228) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.209615 s, 48.9 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 12:37:20 (1788626240) 0+1 records in 0+1 records out 158 bytes copied, 0.0392403 s, 4.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218966) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 12:37:29 (1788626249) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0276272 s, 37.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534866 s, 19.6 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 12:37:49 (1788626269) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0235393 s, 43.5 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 12:38:01 (1788626281) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338757 s, 31.0 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 12:38:08 (1788626288) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 12:38:17 (1788626297) llite.lustre-ffff9af283332800.xattr_cache=1 llite.lustre-ffff9af285452800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:18.000000000 -0400 Modify: 2026-09-05 12:38:18.000000000 -0400 Change: 2026-09-05 12:38:18.000000000 -0400 Birth: 2026-09-05 12:38:19.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:24.000000000 -0400 Modify: 2026-09-05 12:38:25.000000000 -0400 Change: 2026-09-05 12:38:25.000000000 -0400 Birth: 2026-09-05 12:38:25.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:30.000000000 -0400 Modify: 2026-09-05 12:38:31.000000000 -0400 Change: 2026-09-05 12:38:31.000000000 -0400 Birth: 2026-09-05 12:38:31.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:30.000000000 -0400 Modify: 2026-09-05 12:38:31.000000000 -0400 Change: 2026-09-05 12:38:31.000000000 -0400 Birth: 2026-09-05 12:38:31.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:37.000000000 -0400 Modify: 2026-09-05 12:38:38.000000000 -0400 Change: 2026-09-05 12:38:38.000000000 -0400 Birth: 2026-09-05 12:38:38.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:37.000000000 -0400 Modify: 2026-09-05 12:38:38.000000000 -0400 Change: 2026-09-05 12:38:38.000000000 -0400 Birth: 2026-09-05 12:38:38.000000000 -0400 PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 12:38:48 (1788626328) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 12:39:01 (1788626341) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 12:39:16 (1788626356) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 12:39:34 (1788626374) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 12:39:48 (1788626388) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 12:39:51 (1788626391) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 12:40:10 (1788626410) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (21s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 12:40:31 (1788626431) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788626433 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.792715 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (83s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 12:41:54 (1788626514) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788626517 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.47077 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (93s) == sanityn test 60: Verify data_version behaviour ======== 12:43:27 (1788626607) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 12:43:34 (1788626614) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 12:43:40 (1788626620) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 12:43:48 (1788626628) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00641464 s, 10.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 12:43:52 (1788626632) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0107142 s, 6.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 12:43:55 (1788626635) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 12:43:58 (1788626638) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0235698 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 12:44:02 (1788626642) llite.lustre-ffff9af283332800.xattr_cache=1 llite.lustre-ffff9af285452800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 12:44:08 (1788626648) llite.lustre-ffff9af283332800.xattr_cache=1 llite.lustre-ffff9af285452800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 12:44:14 (1788626654) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0129507 s, 79.1 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0144492 s, 70.9 kB/s 231771: get lock1 231768: get lock2 231768: done 231768: get lock1 231771: done 231771: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231771: cannot get lock2: Resource deadlock avoided 231771: exit rc=0 231768: done 231768: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 12:44:26 (1788626666) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0454686 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177773 s, 59.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232352 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 12:44:37 (1788626677) opening files: ........ closing files: ........ PASS 76 (51s) == sanityn test 77a: check FIFO NRS policy =============== 12:45:28 (1788626728) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233318 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273791 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195292 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069018 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219019 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413519 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447281 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149274 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115181 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122925 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600692 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150502 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174611 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773635 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113407 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13209 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197361 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514045 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166286 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628094 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636042 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797437 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732139 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373601 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608421 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495161 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377892 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044335 s, 23.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387362 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707554 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468749 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558706 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613938 s, 17.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=3 removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 12:45:37 (1788626737) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275334 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450063 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530894 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753218 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636552 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913927 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049252 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301856 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875403 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639283 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819583 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166061 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179553 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196957 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122694 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160766 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238265 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289839 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303517 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142371 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205193 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221313 s, 47.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250001 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290586 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312925 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693815 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331647 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365258 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425097 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479624 s, 21.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459206 s, 22.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478883 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568021 s, 18.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226674 s, 74.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025572 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924657 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105321 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103042 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852691 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113374 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089305 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116019 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976053 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132772 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160123 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151223 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174261 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179134 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171248 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239635 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294751 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662978 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112721 s, 93.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399902 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591713 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816876 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573862 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106555 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145407 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531602 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544309 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888453 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588444 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069734 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733747 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997833 s, 10.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 12:45:49 (1788626749) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.240071 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101094 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700857 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762414 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071529 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714797 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032273 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934278 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228183 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552949 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131682 s, 8.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705649 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160304 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633911 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085251 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241671 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271418 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154554 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582403 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480748 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706958 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361945 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581249 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381554 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105466 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410965 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822836 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461627 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745298 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562274 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588107 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589869 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618258 s, 17.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313631 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272341 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293015 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02068 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170291 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120315 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170124 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101631 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179626 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186357 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191916 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184765 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211871 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209377 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164476 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183249 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218973 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928209 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599446 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979171 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491625 s, 21.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735786 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567758 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095541 s, 11.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401339 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101887 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076526 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118975 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950011 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551418 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736168 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602241 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103475 s, 10.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.242507 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328171 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389791 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958051 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036701 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142621 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101916 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912182 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792755 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905874 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855651 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191216 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177742 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888119 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718356 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091169 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131049 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00675575 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366447 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611223 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504295 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899087 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562912 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352189 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112847 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448587 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070262 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104938 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136688 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974665 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235566 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827732 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713994 s, 14.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (13s) == sanityn test 77d: check TRR nrs policy ================ 12:46:02 (1788626762) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183907 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166217 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547026 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504199 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654842 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740783 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609312 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981481 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792521 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070777 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110011 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384055 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101908 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195761 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745619 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107993 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862123 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360375 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313611 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160733 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589155 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445714 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084171 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377567 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879537 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895822 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731622 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662713 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667312 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683232 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280167 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720722 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02261 s, 46.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186139 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483686 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614346 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643314 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079274 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121156 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691589 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703982 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530032 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391912 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351569 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120974 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376198 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120418 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435219 s, 24.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352764 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537756 s, 19.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115354 s, 90.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023494 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504598 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355781 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555556 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285125 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605248 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676342 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442218 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057358 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287712 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728212 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863375 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464001 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632243 s, 16.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666793 s, 15.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222545 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245988 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122804 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405297 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268547 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653467 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13763 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146564 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141812 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125794 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123057 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164525 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121656 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119684 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156342 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162093 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129801 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00655159 s, 160 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628763 s, 167 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124669 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167964 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108238 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883228 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10416 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123645 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668328 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641223 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751417 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602497 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588779 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814643 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517059 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600979 s, 17.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (13s) == sanityn test 77e: check TBF NID nrs policy ============ 12:46:15 (1788626775) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.55@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172333 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286436 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513858 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555696 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859483 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455734 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114498 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756076 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626116 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247014 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144335 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152949 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190307 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137174 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130043 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199756 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180426 s, 5.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362149 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616487 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147915 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394041 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158616 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250341 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234191 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282406 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284782 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305219 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330108 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394754 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381964 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444441 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497508 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567187 s, 18.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228914 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023253 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578691 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626644 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399132 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00831316 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267132 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354097 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324346 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149108 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160837 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689593 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116646 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147433 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422088 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579087 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865788 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583887 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371956 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042044 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439002 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105283 s, 99.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046436 s, 226 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00735472 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959158 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638278 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103439 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098783 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327435 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111785 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026007 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021852 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136201 s, 77.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195905 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378953 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256547 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430148 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225697 s, 46.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694777 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891952 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877107 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13067 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141342 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128386 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132083 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161925 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905082 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169808 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126937 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950189 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392933 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403624 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464603 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426375 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489301 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527911 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728556 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842003 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085698 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124169 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651916 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895494 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685966 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699151 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737992 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150771 s, 7.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190729 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426259 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444943 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215376 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071181 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859382 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694989 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125646 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694957 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139782 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718223 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155549 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140558 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490035 s, 21.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191611 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109041 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174306 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408992 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371768 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322546 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581102 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00880897 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394135 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133353 s, 78.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143328 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190513 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798938 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522158 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239404 s, 438 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478649 s, 219 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447737 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328739 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233966 s, 44.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 12:46:36 (1788626796) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230694 s, 72.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330195 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424881 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885934 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919509 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111601 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105952 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999937 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917905 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103556 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114666 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102727 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133041 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134368 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105984 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108447 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143669 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023534 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301995 s, 34.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395993 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049926 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495897 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200689 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231174 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373892 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372894 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526651 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422425 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560816 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669128 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419054 s, 25.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517668 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528891 s, 19.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.321557 s, 52.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416277 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267569 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886114 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870734 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521538 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138925 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113733 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101431 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1228 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123147 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143634 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157501 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119584 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204122 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102981 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242826 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183917 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266484 s, 393 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511013 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00984954 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435526 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884347 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430584 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111041 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852991 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085587 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675403 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819095 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660012 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329523 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129194 s, 81.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131129 s, 80.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301741 s, 55.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579849 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976404 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175692 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14208 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192731 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152156 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135899 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190715 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236094 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142994 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252158 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258163 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193175 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286987 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263214 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303894 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705405 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572531 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716962 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108162 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544215 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544874 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714721 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078846 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479359 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156532 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136612 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322501 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987242 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101246 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118816 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=5 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.331007 s, 50.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324359 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391653 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725529 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961321 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360675 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600834 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469551 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368979 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877885 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112635 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175395 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143872 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172945 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129122 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194469 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244762 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782627 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444165 s, 236 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290109 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025186 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0023414 s, 448 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266034 s, 394 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246204 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100892 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411084 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119412 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989665 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130384 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974211 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246064 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145998 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109944 s, 9.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (21s) == sanityn test 77g: Change TBF type directly ============ 12:46:58 (1788626818) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (10s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 12:47:07 (1788626827) oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 12:47:18 (1788626838) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 12:47:43 (1788626863) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484955 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667507 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064527 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499207 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428885 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068224 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628613 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888629 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121257 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16052 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130396 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241753 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102517 s, 10.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737011 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244595 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13813 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149997 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af283332800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797294 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152278 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125505 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122211 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837252 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596066 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382218 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088209 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093634 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571136 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495598 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363905 s, 288 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530166 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735764 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797092 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157244 s, 66.7 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.86979 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5985 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (68s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 12:48:51 (1788626931) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26776 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108541 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759973 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100352 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203652 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210328 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163616 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198582 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201348 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195116 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177593 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237635 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299565 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32205 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30824 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2573 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322423 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551242 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493992 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550762 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760275 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812201 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839786 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650371 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591877 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826807 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973609 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104492 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871211 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997326 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125125 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120321 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10858 s, 9.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9152 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9624 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26157 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697392 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850921 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953258 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941514 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17453 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156085 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17231 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119845 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13117 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111764 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983937 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148142 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160654 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905193 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218408 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196347 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792481 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968545 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722407 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690926 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111477 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858077 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108787 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730091 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866636 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861802 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080444 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134568 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101363 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101109 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106288 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131328 s, 8.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9477 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9529 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (150s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 12:51:21 (1788627081) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27822 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576906 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044983 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116132 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853099 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146396 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847575 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1387 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194884 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175835 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825826 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185388 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189844 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210325 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920263 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162594 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189143 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619863 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061016 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493667 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525965 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699185 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512355 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776202 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805449 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671972 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847026 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787379 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741062 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855166 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785637 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788624 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870348 s, 12.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9214 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.931 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26011 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155462 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057192 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114116 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117244 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799953 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104796 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159047 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171847 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157199 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183129 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166366 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155834 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178546 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225271 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235807 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168273 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562887 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456474 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560413 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429615 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061006 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063648 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473594 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484334 s, 21.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047417 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486628 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448417 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506856 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471937 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456691 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775724 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535285 s, 19.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.06 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9768 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (150s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 12:53:52 (1788627232) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.362492 s, 46.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248785 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460076 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402188 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928764 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450632 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111376 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494981 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713006 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143701 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147714 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151009 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155244 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251791 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182716 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203215 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268661 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af283332800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487693 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751001 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101996 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109814 s, 95.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545361 s, 192 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705596 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882045 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10458 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104858 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129245 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606363 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809021 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106601 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751495 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221491 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214104 s, 49.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.75985 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94232 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.229657 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212298 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028201 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543224 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225269 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158765 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724485 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131332 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980088 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936975 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798427 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607951 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905823 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112821 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122535 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716507 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911942 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475683 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128884 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290418 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595436 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499644 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587943 s, 178 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486469 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408148 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382965 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367532 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135402 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151445 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822932 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300001 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025279 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214907 s, 48.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.40576 s, 14.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86872 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401544 s, 41.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222853 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322797 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896243 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836189 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122191 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907941 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729859 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110063 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118798 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127001 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136893 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113516 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124945 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125098 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13185 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136281 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224568 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479714 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420442 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317145 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475739 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428537 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425088 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662274 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053712 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531008 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540898 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572234 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05933 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065481 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681546 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704647 s, 14.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.46248 s, 16.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9981 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.621136 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075188 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530322 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591369 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343861 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117777 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613236 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723407 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165566 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184514 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119938 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109927 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149502 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190836 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127998 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160516 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23245 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647932 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810313 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046878 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533231 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821445 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888151 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814168 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10537 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572452 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779889 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779037 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810549 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830784 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112605 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848078 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915855 s, 11.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6472 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91566 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.04211 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89375 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.919994 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253385 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351393 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584193 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592336 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918676 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600375 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809944 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063289 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134552 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198354 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196949 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189052 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213157 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242754 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270919 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288815 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557959 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766388 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056394 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056303 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692213 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912752 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091774 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714232 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717125 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848114 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952789 s, 11.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871977 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082983 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957067 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900501 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964423 s, 10.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8995 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6583 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.331609 s, 50.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304817 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442419 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512462 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567697 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824848 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115989 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773684 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992925 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134474 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969672 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609872 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744551 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246632 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144107 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20529 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131806 s, 8.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382242 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490718 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064711 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310527 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554597 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495001 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398136 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528207 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693474 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083265 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486175 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659532 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752813 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618363 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652071 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677416 s, 15.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.51516 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87851 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.303927 s, 55.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023463 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395798 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365836 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057838 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622609 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407491 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941912 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577175 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841927 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844627 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874976 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101018 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880588 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101423 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280396 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267008 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035997 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207603 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438389 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045769 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471172 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445845 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492827 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485517 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454305 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517119 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031362 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697025 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593833 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037688 s, 27.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610417 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437797 s, 24.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2766 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86752 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3444 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8979 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (410s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 13:00:41 (1788627641) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.307723 s, 54.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303048 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205314 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321977 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408192 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049392 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779157 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845694 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677966 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935376 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117481 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105472 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106192 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117181 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132528 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123813 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14833 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532881 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134932 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520496 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559401 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217522 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221588 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253697 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290297 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576645 s, 18.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322377 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507768 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343115 s, 30.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476001 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358971 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366845 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410378 s, 25.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.81976 s, 18.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87011 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (46s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 13:01:27 (1788627687) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.351782 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376489 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406549 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106946 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881355 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075659 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104896 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135699 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929565 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15497 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143894 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147409 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143881 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175959 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16427 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213863 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231798 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523526 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450517 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556707 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433663 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498257 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585359 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433963 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700215 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570686 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535928 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491389 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666355 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475254 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567237 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570443 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642569 s, 16.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.51169 s, 16.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85038 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (48s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 13:02:16 (1788627736) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447718 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042829 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369317 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060633 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785599 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150618 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167545 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807897 s, 13.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759429 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160973 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732385 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179087 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16284 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847975 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971096 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932038 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151886 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386324 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195856 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041413 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00268578 s, 390 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00675727 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350585 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954532 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552337 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039046 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649827 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055849 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298427 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271092 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059057 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425224 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121943 s, 86.0 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.29049 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00618 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (43s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 13:02:59 (1788627779) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.490793 s, 34.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440318 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336761 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274503 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13404 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100695 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797755 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123682 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170242 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107215 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145051 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097095 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207177 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241162 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240687 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24891 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292475 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667779 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338205 s, 31.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489031 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232111 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707606 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549776 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694851 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895806 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841246 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840185 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960366 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888821 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771321 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804014 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826664 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857641 s, 12.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6674 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93227 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48265 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93612 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (94s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 13:04:33 (1788627873) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.997715 s, 16.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523934 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360228 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792132 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685683 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927318 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102964 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147735 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229052 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212398 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24748 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245856 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209617 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190866 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21624 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186565 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271914 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197539 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644289 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791814 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627919 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761549 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510676 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053398 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105888 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504539 s, 20.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109497 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492871 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457899 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119008 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571244 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564691 s, 18.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8135 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9425 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (72s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 13:05:46 (1788627946) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.504631 s, 33.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034323 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389188 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886096 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102805 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393485 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180413 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185144 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182209 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201099 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180855 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240076 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173614 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254643 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191586 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185883 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199847 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091715 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351289 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560555 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450618 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051628 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073101 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06731 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753791 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555287 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597751 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702151 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807937 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653124 s, 16.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078485 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120867 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844461 s, 12.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6026 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90278 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg455-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.326743 s, 51.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545387 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546012 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469717 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1468 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10644 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149802 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158335 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150376 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182073 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215178 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203114 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229019 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220014 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235939 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251396 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329875 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067122 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650928 s, 16.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483635 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543738 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435044 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047266 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677358 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535299 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757072 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755575 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928835 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478031 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693412 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856542 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998094 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915226 s, 11.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1411 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0333 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0862 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1937 s, 10.3 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (144s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 13:08:10 (1788628090) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:) Stopping client oleg455-client.virtnet /mnt/lustre opts: Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg455-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg455-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg455-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg455-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg455-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.55,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69584/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.55,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg455-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: kmemleak disabled oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre Starting client oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre Started clients oleg455-client.virtnet: 192.168.204.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 Starting client oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 Started clients oleg455-client.virtnet: 192.168.204.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9af289ac6000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af28a212800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af289ac6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af28a212800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.439947 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782106 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105857 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127598 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149969 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149265 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123914 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22653 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242431 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224125 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28934 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261948 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281422 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32583 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492185 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.537314 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.585633 s, 1.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171018 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150114 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140014 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1999 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13977 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149715 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16456 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153416 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156199 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143214 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14046 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146422 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150721 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17831 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227045 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160171 s, 6.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.89184 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08138 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411963 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379588 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765465 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102626 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151569 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194695 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235326 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244814 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226041 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205769 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232573 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252761 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217996 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2212 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276078 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260195 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263781 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597475 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609886 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426307 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512303 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678581 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378891 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556441 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348155 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787079 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741336 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759779 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734449 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570817 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663812 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072056 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925381 s, 11.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6386 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96803 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (232s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 13:12:02 (1788628322) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 13:12:20 (1788628340) On MGS 192.168.204.155, active = nodemap.active=1 On MGS 192.168.204.155, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.67697 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468764 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894242 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977808 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145789 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236001 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220684 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187674 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221248 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271588 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280998 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270751 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284558 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268072 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302652 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282286 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269937 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298932 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342021 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313011 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309404 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348428 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315387 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316304 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320424 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320873 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32164 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323959 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330188 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335388 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338119 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339616 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351405 s, 3.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43125 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18925 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.155, TBF.id = On MGS 192.168.204.155, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.612212 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852437 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100236 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151269 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193071 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170991 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149567 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150699 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202305 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261639 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122908 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248419 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202041 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266985 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228702 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202329 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281256 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935678 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654927 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691447 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135243 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00659162 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666417 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240937 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390432 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136232 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158715 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136164 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159534 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839728 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135007 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162401 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102534 s, 10.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.22485 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.62137 s, 40.0 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg455-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 On MGS 192.168.204.155, active = nodemap.active=1 On MGS 192.168.204.155, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.51463 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881114 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102455 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242508 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206297 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274746 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292684 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330919 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319437 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372244 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35737 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360085 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358744 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381097 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403664 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34579 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347857 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938427 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05214 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00939757 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557419 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119711 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115131 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12525 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866988 s, 12.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492567 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709779 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108196 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580551 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461704 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236433 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311095 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021877 s, 47.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.32677 s, 12.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96453 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.155, TBF.id = On MGS 192.168.204.155, active = nodemap.active=0 PASS 77kj (177s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:15:17 (1788628517) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:15:27 (1788628527) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg455-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.34713 s, 313 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg455-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0724 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:16:24 (1788628584) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443096 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072522 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573895 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891375 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103271 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154281 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143036 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287303 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136264 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21262 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af289ac6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234558 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152579 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310527 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318167 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345609 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202763 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329667 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588566 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629322 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054705 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489676 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703951 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698127 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730007 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766863 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857275 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849705 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679587 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884487 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1113 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870818 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926434 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759335 s, 13.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg455-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg455-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.52834 s, 16.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0474 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.562861 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772902 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832522 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19137 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171608 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26416 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207051 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305007 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29647 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298141 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327662 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288807 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357118 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345827 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34256 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323298 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334529 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af289ac6000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660288 s, 15.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225601 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609697 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452969 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449321 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639377 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478334 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971317 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508976 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942363 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586128 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653751 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677502 s, 15.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07749 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076436 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944528 s, 11.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg455-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg455-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.54868 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19383 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (90s) == sanityn test 77o: Changing rank should not panic ====== 13:17:54 (1788628674) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:18:07 (1788628687) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (116s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:20:04 (1788628804) mds.MDS.mdt.nrs_policies=tbf oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg455-client: oleg455-server: ssh exited with exit code 36 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg455-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg455-client: oleg455-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (46s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 13:20:50 (1788628850) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 274260 kill md_thread xargs PASS 77r (64s) == sanityn test 77s: Check TBF LRU shrinker ============== 13:21:54 (1788628914) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (73s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:23:08 (1788628988) oleg455-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg455-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg455-client: oleg455-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (12s) == sanityn test 79: xattr: intent error ================== 13:23:19 (1788628999) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935687 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 13:23:22.000000000 -0400 Modify: 2026-09-05 13:23:22.000000000 -0400 Change: 2026-09-05 13:23:22.000000000 -0400 Birth: 2026-09-05 13:23:23.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:23:40 (1788629020) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:23:44 (1788629024) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:23:48 (1788629028) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:23:52 (1788629032) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:23:56 (1788629036) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 13:23:59 (1788629039) - open/close 710 (time 1788629056.19 total 10.01 last 70.92) - open/close 1436 (time 1788629066.19 total 20.02 last 72.57) - open/close 2161 (time 1788629076.19 total 30.02 last 72.47) total: 2500 open/close in 35.28 seconds: 70.86 ops/second snapshot_time 1788629166.253828442 secs.nsecs start_time 1788629043.952629875 secs.nsecs elapsed_time 122.301198567 secs.nsecs open 2505 samples [usecs] 129 26753 4176196 12845513938 close 2505 samples [usecs] 142 7334 930644 675349698 mknod 2501 samples [usecs] 602 26620 3648452 10142605392 unlink 2500 samples [usecs] 782 86102 6413171 82483629117 mkdir 2 samples [usecs] 1039 3076 4115 10541297 rmdir 2 samples [usecs] 29777 37484 67261 2291719985 rename 2500 samples [usecs] 482 61324 16729330 232315539696 getattr 3760 samples [usecs] 10 22334 307289 1236060155 parallel_rename_file 2500 samples [usecs] 482 61324 16729330 232315539696 crossdir_rename 2500 samples [usecs] 482 61324 16729330 232315539696 readpage 36 samples [usecs] 4795 73477 960747 36226804173 PASS 81d (131s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:26:11 (1788629171) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7654 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7654 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:26:18 (1788629178) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:26:22 (1788629182) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 13:26:35 (1788629195) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788629197 Finishing test test1 at 1788629201 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:26:47 (1788629207) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 13:26:50 (1788629210) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:26:54 (1788629214) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:26:58 (1788629218) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5815 0x16b7 0x240000400 1 5775 0x168f 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5776 0x1690 0x280000400 0 5816 0x16b8 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 13:27:15 (1788629235) osc.lustre-OST0000-osc-ffff9af289ac6000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9af28a212800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9af289ac6000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9af28a212800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.253294 s, 32.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.02187 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 291258 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0886817 s, 46.2 kB/s osc.lustre-OST0000-osc-ffff9af289ac6000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af28a212800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af289ac6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af28a212800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 13:27:31 (1788629251) llite.lustre-ffff9af289ac6000.fast_read=0 llite.lustre-ffff9af28a212800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0788204 s, 26.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9af289ac6000.fast_read=1 llite.lustre-ffff9af28a212800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:27:51 (1788629271) llite.lustre-ffff9af289ac6000.fast_read=0 llite.lustre-ffff9af28a212800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0106711 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9af289ac6000.fast_read=1 llite.lustre-ffff9af28a212800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:28:05 (1788629285) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:28:08 (1788629288) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011688 s, 350 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:28:16 (1788629296) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184034 s, 11.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:28:23 (1788629303) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286614 s, 7.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 13:28:32 (1788629312) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0228799 s, 524 kB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 13:28:39 (1788629319) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0247837 s, 165 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 13:28:47 (1788629327) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0113321 s, 361 kB/s PASS 101b (9s) == sanityn test 101c: Discard DoM data on close-unlink === 13:28:56 (1788629336) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0246099 s, 166 kB/s PASS 101c (12s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 13:29:08 (1788629348) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 68 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf68:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 13:29:17 (1788629357) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788629360 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788629364 Finishing test test23 at 1788629365 PASS 103 (17s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:29:34 (1788629374) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 13:29:38 (1788629378) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 13:30:07 (1788629407) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:30:10 (1788629410) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0159309 s, 64.3 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 13:30:12.000000000 -0400 Modify: 2026-09-05 13:30:12.000000000 -0400 Change: 2026-09-05 13:30:12.000000000 -0400 Birth: 2026-09-05 13:30:13.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938223 -rw-r--r-- 1 0 0 1788629413 1788629412 1788629412 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 13:30:12.000000000 -0400 Modify: 2026-09-05 13:30:12.000000000 -0400 Change: 2026-09-05 13:30:12.000000000 -0400 Birth: 2026-09-05 13:30:13.000000000 -0400 1788629412 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 13:30:18 (1788629418) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 13:30:25 (1788629425) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.336204 s, 31.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5783 0x1697 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:30:35 (1788629435) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350247 s, 29.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5825 0x16c1 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7654 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 13:30:50 (1788629450) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0101364 s, 987 kB/s llite.lustre-ffff9af289ac6000.hybrid_io=0 llite.lustre-ffff9af28a212800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.66447 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:31:03 (1788629463) 192.168.204.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg455-client.virtnet /mnt/lustre (opts:) 192.168.204.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg455-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre 192.168.204.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg455-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg455-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: kmemleak disabled oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre2 PASS 109 (47s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:31:51 (1788629511) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 13:31:52.000000000 -0400 Modify: 2026-09-05 13:31:52.000000000 -0400 Change: 2026-09-05 13:31:52.000000000 -0400 Birth: 2026-09-05 13:31:53.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg455-client.virtnet oleg455-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:34:39 (1788629679) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 13:34:44 (1788629684) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 13:34:48 (1788629688) lustre-OST0000-osc-ffff9af2939cc000 active. lustre-OST0001-osc-ffff9af2939cc000 active. lustre-OST0000-osc-ffff9af2939cc000 active. lustre-OST0001-osc-ffff9af2939cc000 active. MGC192.168.204.155@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 13:34:55 (1788629695) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 13:34:59 (1788629699) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 13:35:02 (1788629702) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 13:35:06 (1788629706) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00972749 s, 2.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0914583 s, 224 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 13:35:15 (1788629715) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 13:35:26 (1788629726) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.214 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2743 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2964 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2437 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2747 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3398 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2754 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3101 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4269 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4742 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4344 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4512 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4871 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5753 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5002 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6083 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.7499 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.8145 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af290a78000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af290a78000.ost_server_uuid in IDLE state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 10237 sec ============= 13:36:55 (1788629815) === sanityn: start cleanup 13:36:57 (1788629817) === Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg455-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 13:41:18 (1788630078) ===