-----============= acceptance-small: sanityn ============----- Fri Apr 24 11:22:55 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg159-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 11:23:58 (1777044238) === Starting client oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Mount client oleg159-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Started clients oleg159-client.virtnet: 192.168.201.159@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg159-client.virtnet: executing check_config_client /mnt/lustre oleg159-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg159-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee87d59000.idle_timeout=debug osc.lustre-OST0000-osc-ffff97ee89f26000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee87d59000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee89f26000.idle_timeout=debug disable quota as required oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 11:25:38 (1777044338) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100066 s, 10.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7619] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 11:25:46 (1777044346) PASS 0 (26s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 11:26:13 (1777044373) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (23s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 11:26:36 (1777044396) -rw-r--r-- 1 root root 0 Apr 24 11:26 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (24s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 11:27:00 (1777044420) -rw-r--r-- 1 root root 0 Apr 24 11:27 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (20s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 11:27:21 (1777044441) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (21s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 11:27:42 (1777044462) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (19s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 11:28:01 (1777044481) total 1 -rwxrwxrwx 1 root root 0 Apr 24 11:26 f2a -rwxrwxrwx 1 root root 0 Apr 24 11:27 f2b total 1 -rwxrwxrwx 1 root root 0 Apr 24 11:26 f2a -rwxrwxrwx 1 root root 0 Apr 24 11:27 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0380374 s, 13.5 kB/s PASS 2e (21s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 11:28:22 (1777044502) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (10s) == sanityn test 2g: check blocks update on sync write ==== 11:28:33 (1777044513) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 1.15638 s, 1.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (22s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 11:28:55 (1777044535) PASS 3 (20s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 11:29:16 (1777044556) PASS 4 (22s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 11:29:37 (1777044577) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (24s) == sanityn test 6: remove of open file on other node ============================================================================ 11:30:02 (1777044602) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (21s) == sanityn test 7: remove of open directory on other node ======================================================================= 11:30:24 (1777044624) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (23s) == sanityn test 8: remove of open special file on other node ==================================================================== 11:30:47 (1777044647) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (22s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 11:31:10 (1777044670) PASS 9a (24s) == sanityn test 9b: append to striped sparse file ======== 11:31:34 (1777044694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208604 s, 5.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.228968 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (23s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 11:31:57 (1777044717) 1+0 records in 1+0 records out 1 byte copied, 0.0253647 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0760377 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0386305 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0351486 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.090348 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0535507 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0455839 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0539896 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0572921 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0570653 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0409286 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0404334 s, 0.0 kB/s PASS 10a (27s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 11:32:24 (1777044744) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0775677 s, 39.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.093629 s, 43.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00054323 s, 5.7 MB/s PASS 10b (24s) == sanityn test 11: execution of file opened for write should return error ============================================================== 11:32:48 (1777044768) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (23s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 11:33:11 (1777044791) warning: '-runtime' deprecated, use '-t runtime' instead - link 115 (time 1777044809.04 total 10.04 last 11.45) - link 239 (time 1777044819.05 total 20.05 last 12.39) - link 350 (time 1777044829.06 total 30.06 last 11.09) - link 466 (time 1777044839.07 total 40.07 last 11.59) - link 578 (time 1777044849.16 total 50.16 last 11.10) - link 695 (time 1777044859.22 total 60.22 last 11.63) - link 808 (time 1777044869.25 total 70.25 last 11.27) - link 924 (time 1777044879.27 total 80.27 last 11.58) - link 1035 (time 1777044889.28 total 90.28 last 11.09) total: 1140 link in 100.03 seconds: 11.40 ops/second using seed 2543268985 running for 100 seconds - stat 1093 (time 1777044811 ; total 11 ; last 11) - stat 2280 (time 1777044822 ; total 22 ; last 11) - stat 3316 (time 1777044833 ; total 33 ; last 11) - stat 4316 (time 1777044844 ; total 44 ; last 11) - stat 5222 (time 1777044855 ; total 55 ; last 11) - stat 5750 (time 1777044866 ; total 66 ; last 11) - stat 6313 (time 1777044877 ; total 77 ; last 11) - stat 6895 (time 1777044888 ; total 88 ; last 11) - stat 7336 (time 1777044899 ; total 99 ; last 11) total: 7410 stats in 100 seconds: 74.099998 stats/second - unlinked 0 (time 1777044850 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile1140) error: No such file or directory total: 1139 unlinks in 96 seconds: 11.864583 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21491) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21493) - No such process PASS 12 (172s) == sanityn test 13: test directory page revocation ======= 11:36:04 (1777044964) f13 f13 PASS 13 (33s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 11:36:37 (1777044997) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 22686 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (24s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 11:37:01 (1777045021) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 23269 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (23s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 11:37:25 (1777045045) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23850 Terminated $DIR1/$tdir/sleep 60 PASS 14b (24s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 11:37:50 (1777045070) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24436 Terminated $DIR1/$tdir/sleep 60 PASS 14c (24s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 11:38:15 (1777045095) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 25024 Terminated $DIR1/$tdir/sleep 60 PASS 14d (33s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 11:38:49 (1777045129) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (130s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 11:41:00 (1777045260) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.98903 s, 4.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.61229 s, 5.2 MB/s Chance of close/open is 1 in 50 Seed set to 5274 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x7aaec2 000001[1] 1777045278.697962 trunc from 00000000 to 0x7aaec1 (0x7aaec2 bytes) 000002[1] 1777045279.068563 write 0x1e4da5 thru 0x1e7d2e (0x2f8a bytes) 000005[1] 1777045279.427054 trunc from 0x7aaec2 to 0x340103 (0x46adbe bytes) 000006[1] 1777045279.631606 mapread 0x05a3ef thru 0x0672c4 (0xced6 bytes) 000008[0] 1777045279.872269 write 0x26e9b3 thru 0x275198 (0x67e6 bytes) 000009[1] 1777045280.190121 mapread 0x024757 thru 0x0256e3 (0xf8d bytes) 000011[1] 1777045280.531129 read 0x1a2732 thru 0x1aad7a (0x8649 bytes) 000012[0] 1777045280.709259 mapread 0x0196dd thru 0x020062 (0x6986 bytes) 000013[0] 1777045280.932109 mapwrite 0x67cf39 thru 0x67e69e (0x1766 bytes) 000014[1] 1777045281.556511 mapwrite 0x087a34 thru 0x092f2c (0xb4f9 bytes) 000015[1] 1777045282.471709 mapread 0x2b9ec1 thru 0x2bb831 (0x1971 bytes) 000016[1] 1777045282.776875 write 0x773fbc thru 0x7758de (0x1923 bytes) 000024[1] 1777045283.393327 mapread 0x6b0b46 thru 0x6b3fc5 (0x3480 bytes) 000025[0] 1777045283.654723 trunc from 0x7758df to 0x76e55a (0x7384 bytes) 000026[0] 1777045283.882784 read 0x290181 thru 0x299482 (0x9302 bytes) 000027[1] 1777045284.098441 read 0x15d41a thru 0x163725 (0x630c bytes) 000030[1] 1777045284.376987 read 0x119ec7 thru 0x125067 (0xb1a1 bytes) 000032[1] 1777045284.707949 read 0x4be88e thru 0x4c527b (0x69ee bytes) 000034[0] 1777045284.870517 mapwrite 0x99cc06 thru 0x9a63ee (0x97e9 bytes) 000035[1] 1777045285.693514 trunc from 0x9a63ef to 0x3b520f (0x5f11df bytes) 000037[1] 1777045286.045060 mapread 0x226fbe thru 0x230488 (0x94cb bytes) 000038[0] 1777045286.261864 mapread 0x0820c1 thru 0x091269 (0xf1a9 bytes) 000039[0] 1777045286.562243 mapwrite 0x27f060 thru 0x28c380 (0xd321 bytes) 000042[1] 1777045287.494190 write 0x2c6232 thru 0x2cee41 (0x8c10 bytes) 000044[1] 1777045287.929737 mapwrite 0x965243 thru 0x96ea51 (0x980f bytes) 000047[0] 1777045288.990087 mapread 0x2dd401 thru 0x2e065a (0x325a bytes) 000049[0] 1777045289.449756 read 0x797e14 thru 0x79fdcf (0x7fbc bytes) 000050[0] 1777045289.614205 trunc from 0x96ea52 to 0x5fae3d (0x373c14 bytes) 000054[1] 1777045290.202960 write 0x0b1ca2 thru 0x0b5a5a (0x3db9 bytes) 000056[1] 1777045290.437274 mapread 0x226192 thru 0x22ec4e (0x8abd bytes) 000063[0] 1777045291.038318 trunc from 0x5fae3e to 0x176b80 (0x4842bd bytes) 000066[1] 1777045291.236741 read 0x107905 thru 0x11114a (0x9846 bytes) 000069[0] 1777045291.581047 read 0x0c2c95 thru 0x0c8967 (0x5cd3 bytes) 000070[1] 1777045291.723639 trunc from 0x176b81 to 0x36640a (0x1ef88a bytes) 000072[0] 1777045291.990474 trunc from 0x36640b to 0x08770b (0x2decff bytes) 000073[0] 1777045292.291597 mapwrite 0x3ad071 thru 0x3b7a24 (0xa9b4 bytes) 000074[0] 1777045292.988892 write 0x735930 thru 0x739633 (0x3d04 bytes) 000077[1] 1777045293.150330 write 0x3eba4b thru 0x3f1ebd (0x6473 bytes) 000078[0] 1777045293.352636 write 0x1a3f2f thru 0x1b2a2d (0xeaff bytes) 000082[0] 1777045294.080640 read 0x633911 thru 0x63b13b (0x782b bytes) 000083[0] 1777045294.116040 mapread 0x6da68a thru 0x6e2846 (0x81bd bytes) 000085[0] 1777045294.391906 read 0x2f3492 thru 0x2fe0ca (0xac39 bytes) 000086[1] 1777045294.446752 read 0x5ffe26 thru 0x6037e7 (0x39c2 bytes) 000087[0] 1777045294.698136 trunc from 0x739634 to 0x10953e (0x6300f5 bytes) 000089[1] 1777045295.025427 trunc from 0x10953f to 0x0a8058 (0x614e6 bytes) 000090[0] 1777045295.185213 mapwrite 0x7a471d thru 0x7b134c (0xcc30 bytes) 000092[1] 1777045295.851391 read 0x0aab3b thru 0x0b367d (0x8b43 bytes) 000097[1] 1777045296.512816 mapread 0x4cebbe thru 0x4d3ddd (0x5220 bytes) 000099[0] 1777045297.053377 mapread 0x430559 thru 0x43b79b (0xb243 bytes) 000100[0] 1777045297.172147 mapwrite 0x78ab93 thru 0x78e1dc (0x364a bytes) 000104[0] 1777045298.046647 write 0x30073e thru 0x306ce6 (0x65a9 bytes) 000105[1] 1777045298.311891 write 0x233a7b thru 0x237f3e (0x44c4 bytes) truncating to largest ever: 0x98f57b 000109[0] 1777045298.967205 trunc from 0x7b134d to 0x98f57a (0x1de22e bytes) 000110[1] 1777045299.342574 mapwrite 0x8529dd thru 0x854610 (0x1c34 bytes) 000113[1] 1777045300.242871 mapwrite 0x082e82 thru 0x08ab3e (0x7cbd bytes) 000115[0] 1777045300.974828 trunc from 0x98f57b to 0x2ecf50 (0x6a262a bytes) 000117[1] 1777045301.333717 mapwrite 0x79f28c thru 0x7aaa2e (0xb7a3 bytes) 000118[1] 1777045302.196476 trunc from 0x7aaa2f to 0x4ba078 (0x2f09b6 bytes) 000120[0] 1777045302.463504 trunc from 0x4ba079 to 0x818c21 (0x35eba9 bytes) 000122[0] 1777045302.838525 mapread 0x6d0ccc thru 0x6d5196 (0x44cb bytes) 000126[0] 1777045303.118508 mapwrite 0x8b1465 thru 0x8bd682 (0xc21e bytes) 000127[0] 1777045303.971054 trunc from 0x8bd683 to 0x09698b (0x826cf7 bytes) 000128[0] 1777045304.256255 mapwrite 0x1e03b4 thru 0x1e3da0 (0x39ed bytes) 000131[0] 1777045304.994470 mapwrite 0x484900 thru 0x484d82 (0x483 bytes) 000135[0] 1777045305.654603 trunc from 0x484d83 to 0x85452e (0x3cf7ac bytes) 000138[0] 1777045306.014198 mapread 0x771c96 thru 0x77d15f (0xb4ca bytes) 000139[1] 1777045306.143845 trunc from 0x85452f to 0x59b187 (0x2b93a7 bytes) 000140[1] 1777045306.369542 read 0x295c07 thru 0x2987f3 (0x2bed bytes) 000141[1] 1777045306.540662 trunc from 0x59b188 to 0x84a3fc (0x2af275 bytes) 000142[1] 1777045306.795166 read 0x3028c6 thru 0x312554 (0xfc8f bytes) 000143[0] 1777045306.846690 mapread 0x6144d1 thru 0x61cb2e (0x865e bytes) 000144[0] 1777045307.104925 trunc from 0x84a3fd to 0x5399fd (0x3109ff bytes) 000146[0] 1777045307.599787 write 0x3818a5 thru 0x387839 (0x5f95 bytes) 000147[1] 1777045307.732601 mapread 0x530c5f thru 0x5399fd (0x8d9f bytes) 000153[1] 1777045308.292793 read 0x204106 thru 0x205c78 (0x1b73 bytes) 000154[1] 1777045308.378531 write 0x88cdd7 thru 0x89a720 (0xd94a bytes) 000155[1] 1777045308.408553 read 0x387874 thru 0x388475 (0xc02 bytes) 000156[1] 1777045308.426180 mapread 0x1d5acc thru 0x1e0799 (0xacce bytes) 000157[0] 1777045308.656397 mapread 0x57880f thru 0x586a56 (0xe248 bytes) 000161[0] 1777045309.654582 mapread 0x28ceab thru 0x28ff94 (0x30ea bytes) 000162[0] 1777045309.988266 trunc from 0x89a721 to 0x082c59 (0x817ac7 bytes) 000164[1] 1777045310.373017 read 0x00858d thru 0x00a4f3 (0x1f67 bytes) 000165[1] 1777045310.565226 write 0x40b69e thru 0x419f32 (0xe895 bytes) 000166[1] 1777045310.624693 mapread 0x221d07 thru 0x229acb (0x7dc5 bytes) 000168[1] 1777045311.086866 mapwrite 0x5bb136 thru 0x5c2928 (0x77f3 bytes) 000170[0] 1777045312.134310 mapread 0x43dffe thru 0x446409 (0x840c bytes) 000172[1] 1777045312.369124 trunc from 0x5c2929 to 0x23688e (0x38c09a bytes) 000173[1] 1777045312.529462 mapread 0x163dac thru 0x169cd2 (0x5f27 bytes) 000175[1] 1777045312.757598 mapwrite 0x92a9c8 thru 0x938a41 (0xe07a bytes) 000176[0] 1777045313.516668 write 0x70b2f7 thru 0x70de34 (0x2b3e bytes) 000178[1] 1777045313.862208 mapwrite 0x7507dd thru 0x759c46 (0x946a bytes) 000180[1] 1777045314.509616 read 0x88177f thru 0x884e38 (0x36ba bytes) 000181[1] 1777045314.603216 write 0x4c4f61 thru 0x4cad51 (0x5df1 bytes) 000182[1] 1777045314.692501 write 0x839636 thru 0x83b4af (0x1e7a bytes) 000186[0] 1777045314.936525 write 0x1b62da thru 0x1b6ba4 (0x8cb bytes) 000188[1] 1777045315.127997 mapwrite 0x3186b3 thru 0x321faa (0x98f8 bytes) 000189[0] 1777045315.621845 read 0x4e8d13 thru 0x4efe7a (0x7168 bytes) 000190[1] 1777045315.843433 mapread 0x52f2dc thru 0x52f4bb (0x1e0 bytes) 000191[1] 1777045315.924596 write 0x5a9ff3 thru 0x5b82bd (0xe2cb bytes) 000192[1] 1777045316.003828 write 0x278482 thru 0x287579 (0xf0f8 bytes) 000193[1] 1777045316.060918 mapwrite 0x3ce29b thru 0x3d51c0 (0x6f26 bytes) 000194[0] 1777045316.536974 trunc from 0x938a42 to 0x361e52 (0x5d6bef bytes) 000195[1] 1777045316.987495 mapwrite 0x317f9f thru 0x322aff (0xab61 bytes) 000196[0] 1777045317.654906 read 0x300c1a thru 0x307086 (0x646d bytes) 000197[1] 1777045318.033121 trunc from 0x361e53 to 0x41033d (0xae4eb bytes) 000198[1] 1777045318.532786 write 0x29ee93 thru 0x2a7e8f (0x8ffd bytes) 000200[1] 1777045318.788581 mapread 0x064ef3 thru 0x0738a3 (0xe9b1 bytes) 000201[1] 1777045319.029168 write 0x4f8d02 thru 0x4f8d2d (0x2c bytes) 000202[0] 1777045319.140125 mapread 0x4756d9 thru 0x4781f4 (0x2b1c bytes) 000203[1] 1777045319.389611 trunc from 0x4f8d2e to 0x339d00 (0x1bf02d bytes) 000205[0] 1777045319.554548 write 0x9205b4 thru 0x9236d0 (0x311d bytes) 000208[0] 1777045319.838570 write 0x481a0a thru 0x490d4c (0xf343 bytes) 000209[0] 1777045320.093893 write 0x6cf119 thru 0x6d7a91 (0x8979 bytes) 000210[0] 1777045320.104826 trunc from 0x9236d1 to 0x7729b8 (0x1b0d18 bytes) 000212[1] 1777045320.214370 read 0x1fd9f1 thru 0x1fe6c1 (0xcd1 bytes) 000214[1] 1777045320.671121 mapwrite 0x165fa3 thru 0x16f622 (0x9680 bytes) 000215[0] 1777045321.530597 mapwrite 0x91e3b9 thru 0x92bb15 (0xd75d bytes) 000217[1] 1777045322.486976 mapwrite 0x0b4dc2 thru 0x0c42d2 (0xf511 bytes) 000219[1] 1777045323.561947 mapwrite 0x0f1a28 thru 0x0f4e81 (0x345a bytes) 000220[0] 1777045324.149344 read 0x259d7c thru 0x268938 (0xebbd bytes) 000221[0] 1777045324.257685 trunc from 0x92bb16 to 0x32fa87 (0x5fc08e bytes) 000227[0] 1777045325.067124 mapread 0x2c22f1 thru 0x2cbea3 (0x9bb3 bytes) 000228[1] 1777045325.350202 trunc from 0x32fa88 to 0x8d18f5 (0x5a1e6e bytes) 000229[1] 1777045325.730126 trunc from 0x8d18f6 to 0x227ba9 (0x6a9d4c bytes) 000235[0] 1777045325.914040 mapwrite 0x4136a9 thru 0x41d424 (0x9d7c bytes) 000236[0] 1777045326.606464 mapwrite 0x9295df thru 0x932c59 (0x967b bytes) 000241[1] 1777045327.599802 write 0x29a6aa thru 0x29e29c (0x3bf3 bytes) 000244[1] 1777045327.991237 mapread 0x26fb50 thru 0x279007 (0x94b8 bytes) 000248[0] 1777045328.787920 write 0x679182 thru 0x67b68a (0x2509 bytes) 000250[1] 1777045329.018532 write 0x013be9 thru 0x01e112 (0xa52a bytes) 000251[0] 1777045329.205189 mapwrite 0x4de62b thru 0x4e264e (0x4024 bytes) 000252[0] 1777045329.901889 mapwrite 0x0f1946 thru 0x0f7439 (0x5af4 bytes) 000254[0] 1777045330.925520 mapwrite 0x4fab45 thru 0x50a601 (0xfabd bytes) 000257[1] 1777045331.678404 mapwrite 0x537a46 thru 0x541dc5 (0xa380 bytes) 000260[1] 1777045332.735485 trunc from 0x932c5a to 0x2a2868 (0x6903f1 bytes) 000261[0] 1777045333.096590 mapread 0x1ccdd8 thru 0x1d5e1e (0x9047 bytes) 000262[1] 1777045333.312129 read 0x0f5569 thru 0x0f8541 (0x2fd9 bytes) 000263[0] 1777045333.517190 mapwrite 0x5248db thru 0x528d9b (0x44c1 bytes) 000265[0] 1777045334.411191 mapwrite 0x2bc8c7 thru 0x2c3b3d (0x7277 bytes) 000267[1] 1777045334.981042 trunc from 0x528d9c to 0x05e845 (0x4ca556 bytes) 000269[1] 1777045335.340975 mapread 0x05818c thru 0x05e845 (0x66ba bytes) 000270[1] 1777045335.494943 trunc from 0x05e846 to 0x29ff8d (0x241748 bytes) 000273[0] 1777045335.826850 mapwrite 0x3a762b thru 0x3b5e5c (0xe832 bytes) 000274[0] 1777045336.492356 read 0x334cde thru 0x33bd2f (0x7052 bytes) 000275[0] 1777045336.818669 read 0x055846 thru 0x061b11 (0xc2cc bytes) 000277[0] 1777045337.039017 read 0x059a3f thru 0x05c109 (0x26cb bytes) 000278[0] 1777045337.040569 mapwrite 0x44d3c0 thru 0x454b76 (0x77b7 bytes) 000279[0] 1777045337.767476 trunc from 0x454b77 to 0x34b1ea (0x10998c bytes) 000281[1] 1777045338.045319 trunc from 0x34b1eb to 0x17b89e (0x1cf94c bytes) 000284[1] 1777045338.531626 trunc from 0x17b89f to 0x874728 (0x6f8e8a bytes) 000287[0] 1777045339.040890 trunc from 0x874729 to 0x233801 (0x640f27 bytes) 000290[0] 1777045339.416651 mapwrite 0x67bf5b thru 0x68acf3 (0xed99 bytes) 000291[0] 1777045340.281550 read 0x4e635c thru 0x4f60b9 (0xfd5e bytes) 000292[1] 1777045340.283157 trunc from 0x68acf4 to 0x3b80e5 (0x2d2c0e bytes) 000295[1] 1777045340.617223 mapread 0x08eca1 thru 0x09ca67 (0xddc7 bytes) 000296[1] 1777045341.209838 write 0x2849ff thru 0x287072 (0x2674 bytes) 000297[1] 1777045341.250753 mapread 0x16528d thru 0x168a5e (0x37d2 bytes) 000301[1] 1777045341.460593 write 0x63752d thru 0x646b6e (0xf642 bytes) 000303[0] 1777045341.575781 read 0x01178d thru 0x013fbb (0x282f bytes) 000304[1] 1777045341.756384 read 0x0fce7f thru 0x0ff171 (0x22f3 bytes) 000305[0] 1777045341.792415 read 0x06ddca thru 0x07129c (0x34d3 bytes) 000306[0] 1777045342.205850 write 0x751c7d thru 0x761986 (0xfd0a bytes) 000307[0] 1777045342.357738 mapwrite 0x9737c6 thru 0x973d21 (0x55c bytes) 000311[0] 1777045343.415105 mapread 0x05e6a1 thru 0x06a972 (0xc2d2 bytes) 000312[1] 1777045343.807035 mapwrite 0x81479a thru 0x81bc0b (0x7472 bytes) 000313[0] 1777045344.567031 mapread 0x14f935 thru 0x1577b1 (0x7e7d bytes) 000314[0] 1777045344.935061 read 0x41d27b thru 0x42bf57 (0xecdd bytes) 000317[1] 1777045345.135039 trunc from 0x973d22 to 0x424846 (0x54f4db bytes) 000319[1] 1777045345.637451 mapwrite 0x60a279 thru 0x60b882 (0x160a bytes) 000322[1] 1777045346.063990 write 0x443657 thru 0x44c370 (0x8d1a bytes) 000325[1] 1777045346.349542 mapread 0x0bf176 thru 0x0c705e (0x7ee9 bytes) 000326[0] 1777045346.626233 trunc from 0x60b883 to 0x5e5ef3 (0x2598f bytes) 000328[1] 1777045347.148706 trunc from 0x5e5ef4 to 0x3b9870 (0x22c683 bytes) 000330[0] 1777045347.622408 mapwrite 0x94ce28 thru 0x953253 (0x642c bytes) 000331[0] 1777045348.448173 read 0x93b4a3 thru 0x93f01b (0x3b79 bytes) 000332[1] 1777045348.553043 mapwrite 0x897950 thru 0x89f087 (0x7738 bytes) 000333[1] 1777045349.124477 mapwrite 0x4c069e thru 0x4cf749 (0xf0ac bytes) 000336[0] 1777045349.818892 write 0x40364d thru 0x40890b (0x52bf bytes) 000338[1] 1777045350.055930 trunc from 0x953254 to 0x2635d3 (0x6efc80 bytes) 000339[0] 1777045350.233323 mapwrite 0x2874f7 thru 0x28d3b6 (0x5ec0 bytes) 000340[0] 1777045350.894743 mapread 0x139b1b thru 0x146c42 (0xd128 bytes) 000344[1] 1777045351.217908 mapread 0x16a32d thru 0x16e952 (0x4626 bytes) 000345[1] 1777045351.536125 trunc from 0x28d3b7 to 0x84a096 (0x5bcce0 bytes) 000346[1] 1777045351.742316 mapwrite 0x3758b2 thru 0x37c3ce (0x6b1d bytes) 000347[1] 1777045352.130967 trunc from 0x84a097 to 0x6d32b2 (0x176de4 bytes) 000348[1] 1777045352.312233 write 0x3f7df1 thru 0x405f0a (0xe11a bytes) 000349[0] 1777045352.585813 mapwrite 0x3861b2 thru 0x394ef7 (0xed46 bytes) 000350[1] 1777045353.222529 trunc from 0x6d32b3 to 0x86d5e0 (0x19a32e bytes) 000356[0] 1777045353.937430 mapread 0x03d03a thru 0x046625 (0x95ec bytes) 000357[0] 1777045354.289452 read 0x334f81 thru 0x33e950 (0x99d0 bytes) 000360[1] 1777045354.458223 write 0x736574 thru 0x743a2f (0xd4bc bytes) 000361[0] 1777045354.691029 trunc from 0x86d5e1 to 0x633b94 (0x239a4c bytes) 000363[1] 1777045355.092826 write 0x2946e6 thru 0x2a1c7c (0xd597 bytes) 000365[1] 1777045355.352520 mapread 0x15bccb thru 0x15dad8 (0x1e0e bytes) 000366[1] 1777045355.600322 mapwrite 0x556f95 thru 0x5603f3 (0x945f bytes) 000368[1] 1777045356.267129 read 0x231cd2 thru 0x23e3fe (0xc72d bytes) 000371[0] 1777045356.616014 mapwrite 0x2ca2e8 thru 0x2cdf7a (0x3c93 bytes) 000373[0] 1777045357.428669 write 0x113264 thru 0x116c6b (0x3a08 bytes) 000374[0] 1777045357.540860 read 0x16a0b1 thru 0x172028 (0x7f78 bytes) 000376[0] 1777045357.796668 mapwrite 0x15fb49 thru 0x16481e (0x4cd6 bytes) 000381[1] 1777045358.561633 write 0x8f7763 thru 0x903b07 (0xc3a5 bytes) 000383[0] 1777045358.741160 read 0x27ab52 thru 0x284a52 (0x9f01 bytes) 000387[0] 1777045359.075816 read 0x0e90a5 thru 0x0edfca (0x4f26 bytes) 000390[1] 1777045359.143664 read 0x51d658 thru 0x52a779 (0xd122 bytes) 000392[1] 1777045359.326868 mapread 0x861955 thru 0x8660fb (0x47a7 bytes) 000393[1] 1777045359.507128 read 0x878260 thru 0x885cc7 (0xda68 bytes) 000394[0] 1777045359.650553 read 0x732553 thru 0x739a94 (0x7542 bytes) 000395[1] 1777045359.854537 trunc from 0x903b08 to 0x0f6e97 (0x80cc70 bytes) 000398[1] 1777045360.242174 mapread 0x051d8f thru 0x057f12 (0x6184 bytes) 000399[1] 1777045360.374904 mapwrite 0x5680e7 thru 0x574909 (0xc823 bytes) 000403[0] 1777045361.162392 mapread 0x4ae794 thru 0x4b75da (0x8e47 bytes) 000412[0] 1777045361.954207 trunc from 0x57490a to 0x1f339e (0x38156b bytes) 000414[1] 1777045362.226209 mapwrite 0x7daebb thru 0x7e18ab (0x69f1 bytes) 000420[1] 1777045363.000094 write 0x3fc27e thru 0x400899 (0x461c bytes) 000421[1] 1777045363.082083 mapread 0x0760c9 thru 0x07f592 (0x94ca bytes) 000422[0] 1777045363.163635 trunc from 0x7e18ac to 0x2e1a90 (0x4ffe1b bytes) 000423[0] 1777045363.500975 trunc from 0x2e1a91 to 0x762fe0 (0x481550 bytes) 000424[1] 1777045363.597263 read 0x261bd1 thru 0x265ca9 (0x40d9 bytes) 000425[0] 1777045364.050964 mapread 0x0ea64f thru 0x0ebd9c (0x174e bytes) 000427[0] 1777045364.226543 write 0x4ed64b thru 0x4f005d (0x2a13 bytes) 000428[1] 1777045364.274130 read 0x7222d5 thru 0x729b71 (0x789d bytes) 000429[1] 1777045364.399224 read 0x35b316 thru 0x3668e6 (0xb5d1 bytes) 000430[1] 1777045364.400264 read 0x2efcc4 thru 0x2facfb (0xb038 bytes) 000431[1] 1777045364.416485 mapread 0x123f8c thru 0x126daf (0x2e24 bytes) 000432[0] 1777045364.445660 trunc from 0x762fe1 to 0x0dd0e2 (0x685efe bytes) 000433[1] 1777045364.842193 read 0x00e0b3 thru 0x017468 (0x93b6 bytes) 000434[0] 1777045365.050589 mapwrite 0x35ce85 thru 0x36176a (0x48e6 bytes) 000435[1] 1777045365.904458 write 0x16e9d2 thru 0x17deb6 (0xf4e5 bytes) 000437[0] 1777045366.596664 mapwrite 0x7e9f6f thru 0x7efcfe (0x5d90 bytes) 000438[0] 1777045367.449735 mapread 0x15847e thru 0x167669 (0xf1ec bytes) 000439[1] 1777045367.603825 read 0x5f1cee thru 0x5f8c40 (0x6f53 bytes) 000441[1] 1777045367.712951 mapread 0x7e533e thru 0x7eaed6 (0x5b99 bytes) 000442[0] 1777045368.018909 read 0x663215 thru 0x6692dd (0x60c9 bytes) 000444[1] 1777045368.109364 mapread 0x4242f8 thru 0x42e1d3 (0x9edc bytes) 000447[0] 1777045368.378807 mapread 0x2920da thru 0x29ea48 (0xc96f bytes) 000448[0] 1777045368.561117 trunc from 0x7efcff to 0x5d3822 (0x21c4dc bytes) 000451[1] 1777045368.858723 read 0x3a8e48 thru 0x3b4c51 (0xbe0a bytes) 000454[0] 1777045369.094245 trunc from 0x5d3823 to 0x0b7671 (0x51c1b1 bytes) 000455[1] 1777045369.264072 write 0x790e0a thru 0x795e34 (0x502b bytes) 000459[0] 1777045369.658496 read 0x5a7327 thru 0x5ac11d (0x4df7 bytes) 000460[0] 1777045369.856133 write 0x2ca3de thru 0x2d1dac (0x79cf bytes) 000461[1] 1777045370.002643 read 0x4d499f thru 0x4ddf02 (0x9564 bytes) 000462[1] 1777045370.067049 trunc from 0x795e35 to 0x1dd655 (0x5b87df bytes) 000463[1] 1777045370.401152 trunc from 0x1dd656 to 0x41e628 (0x240fd3 bytes) 000465[0] 1777045370.582768 trunc from 0x41e629 to 0x5ea84b (0x1cc223 bytes) 000468[0] 1777045370.881795 read 0x03cf09 thru 0x03fa05 (0x2afd bytes) 000478[0] 1777045371.728669 write 0x05627c thru 0x05d4e6 (0x726b bytes) 000480[0] 1777045371.998404 read 0x5e6ecb thru 0x5ea84b (0x3981 bytes) 000482[1] 1777045372.066456 trunc from 0x5ea84c to 0x60cb9c (0x22351 bytes) 000483[1] 1777045372.514762 write 0x945cc5 thru 0x946786 (0xac2 bytes) 000484[0] 1777045372.565963 mapwrite 0x24e12a thru 0x25af5b (0xce32 bytes) 000486[0] 1777045373.370831 read 0x0e7aba thru 0x0ee3c9 (0x6910 bytes) 000488[1] 1777045373.601257 mapread 0x10244f thru 0x106e87 (0x4a39 bytes) 000492[1] 1777045374.067480 read 0x81135f thru 0x8180c5 (0x6d67 bytes) 000493[0] 1777045374.280397 read 0x28eb86 thru 0x29d97c (0xedf7 bytes) 000494[1] 1777045374.466448 trunc from 0x946787 to 0x50baaa (0x43acdc bytes) 000495[0] 1777045374.750099 trunc from 0x50baab to 0x4a017e (0x6b92c bytes) 000496[1] 1777045375.052475 mapwrite 0x60514a thru 0x60ec44 (0x9afb bytes) 000499[1] 1777045375.759459 mapwrite 0x24b74a thru 0x257a38 (0xc2ef bytes) 000500[0] 1777045376.267537 mapread 0x531db9 thru 0x53fb80 (0xddc8 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 5376 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1777045380.140358 mapwrite 0x281000 thru 0x28ef49 (0xdf4a bytes) 000003[0] 1777045380.769134 write_OD 0x8b8000 thru 0x8bdfff (0x6000 bytes) 000004[1] 1777045381.118951 mapwrite 0x0ba000 thru 0x0c67c9 (0xc7ca bytes) 000005[1] 1777045381.687719 write_OD 0x687000 thru 0x693fff (0xd000 bytes) 000007[1] 1777045381.982803 write_OD 0x96c000 thru 0x96cfff (0x1000 bytes) 000008[0] 1777045382.340458 mapread 0x7fa000 thru 0x7fb5c0 (0x15c1 bytes) truncating to largest ever: 0x39c4c2 000010[0] 1777045382.435343 trunc from 0x96d000 to 0x39c4c1 (0x5d0b3e bytes) 000012[1] 1777045382.701849 write_OD 0x5b6000 thru 0x5bffff (0xa000 bytes) 000014[0] 1777045382.985232 read_OD 0x4e9000 thru 0x4f6fff (0xe000 bytes) truncating to largest ever: 0x960ba2 000015[0] 1777045383.029511 trunc from 0x5c0000 to 0x960ba1 (0x3a0ba2 bytes) 000017[1] 1777045383.143314 write_OD 0x243000 thru 0x248fff (0x6000 bytes) 000020[0] 1777045383.286471 read_OD 0x449000 thru 0x457fff (0xf000 bytes) 000023[1] 1777045383.384645 mapread 0x911000 thru 0x912789 (0x178a bytes) 000025[1] 1777045383.512961 mapread 0x609000 thru 0x6169cd (0xd9ce bytes) 000026[1] 1777045383.682275 read_OD 0x429000 thru 0x42afff (0x2000 bytes) 000028[1] 1777045383.705025 write_OD 0x2ae000 thru 0x2b1fff (0x4000 bytes) 000029[1] 1777045384.008901 mapread 0x929000 thru 0x9350d7 (0xc0d8 bytes) 000031[1] 1777045384.183959 mapread 0x635000 thru 0x63ef8f (0x9f90 bytes) 000032[0] 1777045384.282718 trunc from 0x960ba2 to 0x43852b (0x528676 bytes) 000033[1] 1777045384.495536 trunc from 0x43852c to 0x7c3024 (0x38aaf9 bytes) 000039[1] 1777045384.618042 write_OD 0x8cf000 thru 0x8d1fff (0x3000 bytes) 000040[1] 1777045384.710359 mapwrite 0x044000 thru 0x045a69 (0x1a6a bytes) 000041[1] 1777045385.024981 write_OD 0x034000 thru 0x03afff (0x7000 bytes) 000043[1] 1777045385.318955 write_OD 0x856000 thru 0x857fff (0x2000 bytes) 000045[0] 1777045385.600546 mapread 0x011000 thru 0x020aa4 (0xfaa5 bytes) 000046[1] 1777045385.832903 write_OD 0x396000 thru 0x39afff (0x5000 bytes) 000048[0] 1777045386.183671 trunc from 0x8d2000 to 0x753ed0 (0x17e12f bytes) 000049[0] 1777045386.353399 read_OD 0x52a000 thru 0x534fff (0xb000 bytes) 000050[1] 1777045386.451494 trunc from 0x753ed1 to 0x5e6184 (0x16dd4c bytes) 000052[1] 1777045386.731488 trunc from 0x5e6185 to 0x01cbae (0x5c95d6 bytes) 000053[0] 1777045386.834976 trunc from 0x01cbaf to 0x46e9e0 (0x451e32 bytes) 000054[1] 1777045387.002703 read_OD 0x2fc000 thru 0x306fff (0xb000 bytes) 000055[0] 1777045387.190662 read_OD 0x40b000 thru 0x417fff (0xd000 bytes) 000056[1] 1777045387.225130 mapwrite 0x170000 thru 0x178f1a (0x8f1b bytes) 000061[1] 1777045388.032645 write_OD 0x88e000 thru 0x897fff (0xa000 bytes) 000062[1] 1777045388.236243 mapwrite 0x947000 thru 0x94b2a7 (0x42a8 bytes) 000064[0] 1777045388.808098 write_OD 0x906000 thru 0x909fff (0x4000 bytes) 000065[1] 1777045389.075860 write_OD 0x2e3000 thru 0x2eefff (0xc000 bytes) 000067[0] 1777045389.501968 mapread 0x84b000 thru 0x850714 (0x5715 bytes) 000068[0] 1777045389.597582 mapwrite 0x99e000 thru 0x9a1126 (0x3127 bytes) 000069[0] 1777045389.937385 mapwrite 0x0de000 thru 0x0edd3d (0xfd3e bytes) 000073[0] 1777045390.620752 mapread 0x1f9000 thru 0x2023ba (0x93bb bytes) 000074[1] 1777045390.779970 mapwrite 0x9cd000 thru 0x9d195e (0x495f bytes) 000076[0] 1777045391.237591 trunc from 0x9d195f to 0x485f5b (0x54ba03 bytes) 000077[1] 1777045391.336384 mapread 0x28a000 thru 0x28e702 (0x4703 bytes) 000079[0] 1777045391.666130 trunc from 0x485f5c to 0x958f5e (0x4d3003 bytes) 000082[0] 1777045392.049036 trunc from 0x958f5f to 0x44cdff (0x50c15f bytes) 000084[1] 1777045392.247353 read_OD 0x387000 thru 0x38cfff (0x6000 bytes) 000086[0] 1777045392.408965 read_OD 0x0e4000 thru 0x0ebfff (0x8000 bytes) 000087[0] 1777045392.461804 mapread 0x016000 thru 0x0176d5 (0x16d6 bytes) 000088[0] 1777045392.513585 trunc from 0x44ce00 to 0x1f843f (0x2549c0 bytes) 000089[1] 1777045392.633966 read_OD 0x040000 thru 0x049fff (0xa000 bytes) 000090[1] 1777045392.808744 mapwrite 0x66d000 thru 0x66e1ba (0x11bb bytes) 000094[0] 1777045393.553588 write_OD 0x07a000 thru 0x07afff (0x1000 bytes) 000097[1] 1777045393.896834 mapwrite 0x60b000 thru 0x61495d (0x995e bytes) 000098[1] 1777045394.443413 mapread 0x31c000 thru 0x327935 (0xb936 bytes) 000104[1] 1777045395.068735 write_OD 0x4c8000 thru 0x4ccfff (0x5000 bytes) 000107[0] 1777045395.342145 mapread 0x517000 thru 0x51bdc3 (0x4dc4 bytes) 000108[0] 1777045395.460688 write_OD 0x6ba000 thru 0x6bbfff (0x2000 bytes) 000112[0] 1777045395.819592 trunc from 0x6bc000 to 0x6a3366 (0x18c99 bytes) 000114[1] 1777045396.247681 trunc from 0x6a3367 to 0x377498 (0x32bece bytes) 000115[1] 1777045396.344517 mapread 0x1af000 thru 0x1b17bf (0x27c0 bytes) 000118[1] 1777045396.618692 mapread 0x019000 thru 0x01cbc9 (0x3bca bytes) 000119[0] 1777045396.831496 mapwrite 0x674000 thru 0x678e6a (0x4e6b bytes) 000120[0] 1777045397.366416 read_OD 0x673000 thru 0x677fff (0x5000 bytes) 000125[1] 1777045397.455147 write_OD 0x369000 thru 0x36bfff (0x3000 bytes) 000126[1] 1777045397.672839 write_OD 0x993000 thru 0x9a1fff (0xf000 bytes) 000127[1] 1777045398.015495 write_OD 0x1a6000 thru 0x1a9fff (0x4000 bytes) 000130[0] 1777045398.285985 mapwrite 0x462000 thru 0x467a48 (0x5a49 bytes) 000133[0] 1777045398.708945 trunc from 0x9a2000 to 0x3302b7 (0x671d48 bytes) 000135[0] 1777045398.862234 read_OD 0x093000 thru 0x093fff (0x1000 bytes) 000137[0] 1777045398.938854 mapwrite 0x5e3000 thru 0x5e5ee5 (0x2ee6 bytes) 000138[1] 1777045399.795563 trunc from 0x5e5ee6 to 0x94e290 (0x3683ab bytes) 000143[0] 1777045400.095114 read_OD 0x8b4000 thru 0x8c2fff (0xf000 bytes) 000144[0] 1777045400.273226 trunc from 0x94e291 to 0x75a197 (0x1f40f9 bytes) 000145[0] 1777045400.466378 trunc from 0x75a198 to 0x5f772a (0x162a6d bytes) 000149[1] 1777045400.788603 mapwrite 0x023000 thru 0x02fc3a (0xcc3b bytes) 000150[1] 1777045401.238079 mapwrite 0x8b5000 thru 0x8b7dcb (0x2dcc bytes) 000152[1] 1777045401.566467 read_OD 0x38c000 thru 0x396fff (0xb000 bytes) 000155[0] 1777045401.658231 read_OD 0x57e000 thru 0x580fff (0x3000 bytes) 000157[0] 1777045401.694200 read_OD 0x0f5000 thru 0x103fff (0xf000 bytes) 000158[0] 1777045401.732118 read_OD 0x171000 thru 0x17afff (0xa000 bytes) 000163[0] 1777045401.769146 trunc from 0x8b7dcc to 0x8dbaff (0x23d34 bytes) 000164[1] 1777045401.972702 trunc from 0x8dbb00 to 0x843826 (0x982d9 bytes) 000167[0] 1777045402.332115 trunc from 0x843827 to 0x825be6 (0x1dc40 bytes) 000169[0] 1777045402.565025 mapwrite 0x539000 thru 0x53a986 (0x1987 bytes) 000171[0] 1777045402.917898 mapread 0x7c7000 thru 0x7c91a8 (0x21a9 bytes) 000173[1] 1777045403.128873 mapread 0x07f000 thru 0x07ff5a (0xf5b bytes) 000175[1] 1777045403.243892 mapread 0x48a000 thru 0x490693 (0x6694 bytes) 000176[1] 1777045403.358496 read_OD 0x62f000 thru 0x63bfff (0xd000 bytes) 000178[1] 1777045403.417270 read_OD 0x376000 thru 0x376fff (0x1000 bytes) 000180[1] 1777045403.449037 write_OD 0x034000 thru 0x042fff (0xf000 bytes) 000184[0] 1777045403.677714 read_OD 0x1fd000 thru 0x202fff (0x6000 bytes) 000185[1] 1777045403.732227 read_OD 0x66b000 thru 0x66bfff (0x1000 bytes) 000186[1] 1777045403.766591 mapwrite 0x099000 thru 0x0a647e (0xd47f bytes) 000188[1] 1777045404.508341 mapwrite 0x291000 thru 0x29335f (0x2360 bytes) 000190[0] 1777045404.872395 mapread 0x6df000 thru 0x6dfba0 (0xba1 bytes) 000193[0] 1777045404.975082 write_OD 0x50b000 thru 0x50cfff (0x2000 bytes) 000194[0] 1777045405.218066 write_OD 0x76b000 thru 0x772fff (0x8000 bytes) 000202[0] 1777045405.491503 write_OD 0x3c3000 thru 0x3cffff (0xd000 bytes) 000203[1] 1777045405.747524 mapwrite 0x0b0000 thru 0x0b3738 (0x3739 bytes) 000204[0] 1777045406.215428 mapwrite 0x064000 thru 0x070cbf (0xccc0 bytes) 000205[0] 1777045406.805763 read_OD 0x158000 thru 0x158fff (0x1000 bytes) 000206[0] 1777045406.877531 mapread 0x1de000 thru 0x1e137f (0x3380 bytes) 000208[1] 1777045407.036606 mapwrite 0x1ed000 thru 0x1f59f0 (0x89f1 bytes) 000212[1] 1777045407.803817 mapwrite 0x8cd000 thru 0x8d0aa2 (0x3aa3 bytes) 000216[1] 1777045408.711476 trunc from 0x8d0aa3 to 0x5216b8 (0x3af3ea bytes) 000219[0] 1777045408.996982 write_OD 0x695000 thru 0x6a2fff (0xe000 bytes) 000222[1] 1777045409.230108 trunc from 0x6a3000 to 0x545281 (0x15dd7e bytes) 000223[0] 1777045409.376870 trunc from 0x545282 to 0x568803 (0x23582 bytes) 000225[0] 1777045409.584559 mapwrite 0x742000 thru 0x74bbc1 (0x9bc2 bytes) 000228[1] 1777045410.231156 read_OD 0x4f6000 thru 0x501fff (0xc000 bytes) 000233[0] 1777045410.453401 mapread 0x5cf000 thru 0x5d7415 (0x8416 bytes) 000237[0] 1777045410.563874 write_OD 0x297000 thru 0x2a1fff (0xb000 bytes) 000238[0] 1777045410.838569 trunc from 0x74bbc2 to 0x5ffc41 (0x14bf80 bytes) 000239[1] 1777045410.969847 mapread 0x40e000 thru 0x41ce1f (0xee20 bytes) 000240[0] 1777045411.117805 mapread 0x1b6000 thru 0x1bda04 (0x7a05 bytes) 000242[1] 1777045411.206585 mapwrite 0x190000 thru 0x198eea (0x8eeb bytes) 000243[1] 1777045411.697965 mapread 0x028000 thru 0x02e95c (0x695d bytes) 000246[0] 1777045412.020016 trunc from 0x5ffc42 to 0x332ae6 (0x2cd15b bytes) 000250[0] 1777045412.384016 write_OD 0x9ce000 thru 0x9dafff (0xd000 bytes) 000255[1] 1777045412.605552 mapread 0x184000 thru 0x18fda2 (0xbda3 bytes) 000256[0] 1777045412.703581 trunc from 0x9db000 to 0x36cb5a (0x66e4a5 bytes) 000257[0] 1777045412.863830 write_OD 0x1d1000 thru 0x1dafff (0xa000 bytes) 000258[1] 1777045413.068060 mapwrite 0x367000 thru 0x368652 (0x1653 bytes) 000259[0] 1777045413.342031 mapread 0x2fa000 thru 0x2feb4c (0x4b4d bytes) 000261[0] 1777045413.526725 write_OD 0x890000 thru 0x897fff (0x8000 bytes) 000262[0] 1777045413.739493 mapread 0x7f8000 thru 0x7f8e31 (0xe32 bytes) 000263[0] 1777045413.766200 read_OD 0x71c000 thru 0x729fff (0xe000 bytes) 000264[0] 1777045413.794069 mapread 0x2ce000 thru 0x2d3d7c (0x5d7d bytes) 000265[1] 1777045413.865066 mapread 0x746000 thru 0x755dc1 (0xfdc2 bytes) 000267[1] 1777045413.991998 read_OD 0x08b000 thru 0x08cfff (0x2000 bytes) 000268[1] 1777045414.012019 mapwrite 0x550000 thru 0x5545ef (0x45f0 bytes) 000269[1] 1777045414.368079 read_OD 0x13e000 thru 0x14bfff (0xe000 bytes) 000270[1] 1777045414.484948 read_OD 0x5f8000 thru 0x5f8fff (0x1000 bytes) 000272[1] 1777045414.540049 mapwrite 0x268000 thru 0x27767f (0xf680 bytes) 000274[1] 1777045415.043654 mapwrite 0x676000 thru 0x679f61 (0x3f62 bytes) truncating to largest ever: 0x9614ed 000275[1] 1777045415.285509 trunc from 0x898000 to 0x9614ec (0xc94ed bytes) 000278[1] 1777045415.369967 write_OD 0x608000 thru 0x613fff (0xc000 bytes) 000279[1] 1777045415.613782 trunc from 0x9614ed to 0x1bd9ae (0x7a3b3e bytes) 000280[0] 1777045415.808992 mapread 0x0da000 thru 0x0e1f27 (0x7f28 bytes) 000282[0] 1777045416.048364 trunc from 0x1bd9af to 0x82ed51 (0x6713a3 bytes) 000284[1] 1777045416.225389 trunc from 0x82ed52 to 0x32f161 (0x4ffbf0 bytes) 000285[1] 1777045416.393958 mapwrite 0x5b0000 thru 0x5b8074 (0x8075 bytes) 000287[1] 1777045416.899176 mapwrite 0x441000 thru 0x44faa7 (0xeaa8 bytes) 000288[1] 1777045417.590717 trunc from 0x5b8075 to 0x7453f9 (0x18d385 bytes) 000290[1] 1777045417.848417 mapread 0x608000 thru 0x608237 (0x238 bytes) 000291[1] 1777045417.887247 write_OD 0x311000 thru 0x31efff (0xe000 bytes) 000292[1] 1777045418.167488 read_OD 0x2f7000 thru 0x302fff (0xc000 bytes) 000293[1] 1777045418.246907 read_OD 0x493000 thru 0x494fff (0x2000 bytes) 000294[1] 1777045418.262018 read_OD 0x524000 thru 0x528fff (0x5000 bytes) 000295[1] 1777045418.303095 read_OD 0x3ad000 thru 0x3b9fff (0xd000 bytes) 000296[1] 1777045418.374373 trunc from 0x7453fa to 0x4e257d (0x262e7c bytes) 000298[1] 1777045418.788267 mapread 0x081000 thru 0x08425a (0x325b bytes) 000303[1] 1777045419.035643 write_OD 0x066000 thru 0x06dfff (0x8000 bytes) 000304[1] 1777045419.270189 trunc from 0x4e257e to 0x2c9625 (0x218f58 bytes) 000306[1] 1777045419.413813 trunc from 0x2c9626 to 0x4b27db (0x1e91b6 bytes) 000308[1] 1777045419.528574 mapwrite 0x711000 thru 0x71ffe3 (0xefe4 bytes) 000309[1] 1777045419.894689 trunc from 0x71ffe4 to 0x47b38b (0x2a4c58 bytes) 000310[1] 1777045419.965524 mapwrite 0x7bb000 thru 0x7c43fd (0x93fe bytes) 000313[0] 1777045420.661125 mapwrite 0x73a000 thru 0x73e792 (0x4793 bytes) 000314[1] 1777045421.356821 write_OD 0x53e000 thru 0x544fff (0x7000 bytes) 000315[0] 1777045421.620615 mapwrite 0x92a000 thru 0x933725 (0x9726 bytes) 000317[0] 1777045422.235645 mapwrite 0x6e7000 thru 0x6f66d8 (0xf6d9 bytes) 000318[0] 1777045422.730335 mapwrite 0x28a000 thru 0x28f750 (0x5751 bytes) 000319[1] 1777045423.269405 mapwrite 0x36c000 thru 0x375033 (0x9034 bytes) 000320[0] 1777045423.927100 mapread 0x0d1000 thru 0x0da395 (0x9396 bytes) 000322[0] 1777045424.280566 mapread 0x326000 thru 0x32d2e2 (0x72e3 bytes) 000323[0] 1777045424.383246 read_OD 0x0ee000 thru 0x0fbfff (0xe000 bytes) 000325[1] 1777045424.424182 mapread 0x3a5000 thru 0x3a87fa (0x37fb bytes) 000326[1] 1777045424.550858 mapwrite 0x27e000 thru 0x2858e5 (0x78e6 bytes) 000327[0] 1777045425.133050 mapread 0x743000 thru 0x75072d (0xd72e bytes) 000328[1] 1777045425.424312 read_OD 0x836000 thru 0x841fff (0xc000 bytes) 000331[0] 1777045425.526092 trunc from 0x933726 to 0x301944 (0x631de1 bytes) 000332[1] 1777045425.670342 read_OD 0x1d8000 thru 0x1e6fff (0xf000 bytes) 000333[1] 1777045425.832715 mapread 0x026000 thru 0x0334cd (0xd4ce bytes) 000335[0] 1777045425.982840 read_OD 0x2c8000 thru 0x2d4fff (0xd000 bytes) 000339[0] 1777045426.052324 trunc from 0x301945 to 0x32e54d (0x2cc09 bytes) 000341[1] 1777045426.324671 mapwrite 0x3ca000 thru 0x3ca2aa (0x2ab bytes) 000342[1] 1777045426.634700 mapwrite 0x2a5000 thru 0x2a5cdd (0xcde bytes) 000346[0] 1777045427.009827 mapwrite 0x11e000 thru 0x12aacf (0xcad0 bytes) 000347[0] 1777045427.955810 mapwrite 0x738000 thru 0x73ea93 (0x6a94 bytes) 000348[1] 1777045428.597659 mapwrite 0x37e000 thru 0x3815d8 (0x35d9 bytes) 000349[0] 1777045428.939374 trunc from 0x73ea94 to 0x22fd67 (0x50ed2c bytes) 000352[0] 1777045429.120762 mapread 0x06c000 thru 0x070781 (0x4782 bytes) 000353[0] 1777045429.415494 mapwrite 0x2cb000 thru 0x2da95e (0xf95f bytes) 000355[0] 1777045430.326465 trunc from 0x2da95f to 0x1eaaa5 (0xefeb9 bytes) 000356[1] 1777045430.472446 read_OD 0x1d1000 thru 0x1dafff (0xa000 bytes) 000357[1] 1777045430.660948 write_OD 0x047000 thru 0x04dfff (0x7000 bytes) 000358[1] 1777045430.912345 mapwrite 0x3cd000 thru 0x3d5511 (0x8512 bytes) 000359[0] 1777045431.734003 mapwrite 0x7d2000 thru 0x7d40b9 (0x20ba bytes) 000360[0] 1777045432.302419 mapwrite 0x951000 thru 0x953c8f (0x2c90 bytes) 000364[1] 1777045433.369098 write_OD 0x8e4000 thru 0x8ecfff (0x9000 bytes) 000365[1] 1777045433.765694 mapread 0x94a000 thru 0x94af50 (0xf51 bytes) 000366[0] 1777045434.113465 mapread 0x063000 thru 0x06842e (0x542f bytes) 000367[1] 1777045434.590681 mapwrite 0x89a000 thru 0x89be92 (0x1e93 bytes) 000370[1] 1777045435.156979 trunc from 0x953c90 to 0x891a09 (0xc2286 bytes) 000371[1] 1777045435.409626 write_OD 0x86b000 thru 0x878fff (0xe000 bytes) 000373[1] 1777045435.806066 write_OD 0x9f5000 thru 0x9fefff (0xa000 bytes) 000377[1] 1777045436.282571 trunc from 0x9ff000 to 0x036f03 (0x9c80fc bytes) 000379[1] 1777045436.638521 mapwrite 0x2ac000 thru 0x2ae80c (0x280d bytes) 000384[0] 1777045437.332626 mapwrite 0x76d000 thru 0x777b27 (0xab28 bytes) 000388[0] 1777045438.510188 trunc from 0x777b28 to 0x4d222d (0x2a58fa bytes) 000389[0] 1777045438.616104 write_OD 0x313000 thru 0x314fff (0x2000 bytes) 000390[1] 1777045438.934980 mapread 0x37c000 thru 0x384c93 (0x8c94 bytes) 000391[1] 1777045439.174856 mapwrite 0x236000 thru 0x23e888 (0x8889 bytes) 000392[1] 1777045439.894129 write_OD 0x713000 thru 0x71ffff (0xd000 bytes) 000393[1] 1777045440.171540 mapwrite 0x920000 thru 0x921dcf (0x1dd0 bytes) 000396[1] 1777045440.872450 mapread 0x7bc000 thru 0x7c7a33 (0xba34 bytes) 000399[0] 1777045441.325508 mapwrite 0x7a7000 thru 0x7b01ca (0x91cb bytes) 000402[0] 1777045442.223919 write_OD 0x58c000 thru 0x597fff (0xc000 bytes) 000405[1] 1777045442.584111 trunc from 0x921dd0 to 0x14198d (0x7e0442 bytes) 000407[0] 1777045442.852577 trunc from 0x14198e to 0x2dbb86 (0x19a1f9 bytes) 000414[0] 1777045443.450936 write_OD 0x7f3000 thru 0x7f7fff (0x5000 bytes) 000416[1] 1777045443.904182 mapwrite 0x13c000 thru 0x13da6c (0x1a6d bytes) 000417[0] 1777045444.419978 read_OD 0x691000 thru 0x691fff (0x1000 bytes) 000418[0] 1777045444.574876 mapread 0x2b7000 thru 0x2c4436 (0xd437 bytes) 000423[0] 1777045444.893972 mapread 0x3a9000 thru 0x3b6154 (0xd155 bytes) 000425[1] 1777045444.970173 read_OD 0x176000 thru 0x181fff (0xc000 bytes) 000429[0] 1777045445.041175 mapread 0x7e0000 thru 0x7ec640 (0xc641 bytes) 000432[0] 1777045445.257245 mapwrite 0x46c000 thru 0x47a62e (0xe62f bytes) 000433[0] 1777045445.929456 write_OD 0x05f000 thru 0x061fff (0x3000 bytes) truncating to largest ever: 0x9f6f1e 000434[0] 1777045446.276532 trunc from 0x7f8000 to 0x9f6f1d (0x1fef1e bytes) 000436[0] 1777045446.479701 trunc from 0x9f6f1e to 0x14709f (0x8afe7e bytes) 000441[1] 1777045447.010474 trunc from 0x1470a0 to 0x6385ac (0x4f150d bytes) 000445[0] 1777045447.606690 read_OD 0x0ea000 thru 0x0f8fff (0xf000 bytes) 000446[1] 1777045447.736944 write_OD 0x8ef000 thru 0x8effff (0x1000 bytes) 000451[0] 1777045448.096286 mapwrite 0x31b000 thru 0x326bba (0xbbbb bytes) 000452[0] 1777045448.956461 mapwrite 0x12c000 thru 0x137f3e (0xbf3f bytes) 000453[1] 1777045449.514511 write_OD 0x4ef000 thru 0x4effff (0x1000 bytes) 000455[0] 1777045449.735639 trunc from 0x8f0000 to 0x80733c (0xe8cc3 bytes) 000456[0] 1777045449.913782 read_OD 0x31d000 thru 0x325fff (0x9000 bytes) 000461[0] 1777045450.084723 read_OD 0x3bf000 thru 0x3c7fff (0x9000 bytes) 000466[1] 1777045450.159942 mapread 0x0c5000 thru 0x0cf2e0 (0xa2e1 bytes) 000467[1] 1777045450.377499 write_OD 0x426000 thru 0x428fff (0x3000 bytes) 000468[1] 1777045450.765398 trunc from 0x80733d to 0x3133e2 (0x4f3f5a bytes) 000469[1] 1777045451.047546 mapread 0x190000 thru 0x193713 (0x3714 bytes) 000477[0] 1777045451.545469 trunc from 0x3133e3 to 0x04e513 (0x2c4ecf bytes) 000480[1] 1777045451.964482 write_OD 0x8a2000 thru 0x8aefff (0xd000 bytes) 000484[0] 1777045452.395910 mapread 0x5d2000 thru 0x5dd9dc (0xb9dd bytes) 000486[1] 1777045452.560668 read_OD 0x281000 thru 0x28afff (0xa000 bytes) 000488[0] 1777045452.619788 mapread 0x3f9000 thru 0x3f9cee (0xcef bytes) 000490[0] 1777045452.649725 write_OD 0x021000 thru 0x02efff (0xe000 bytes) 000492[1] 1777045453.170337 mapwrite 0x210000 thru 0x211198 (0x1199 bytes) 000493[1] 1777045453.601816 mapread 0x5a8000 thru 0x5b6c8f (0xec90 bytes) 000494[1] 1777045453.920417 read_OD 0x448000 thru 0x44efff (0x7000 bytes) 000495[1] 1777045454.051514 mapwrite 0x27f000 thru 0x281c31 (0x2c32 bytes) 000498[0] 1777045454.707090 mapwrite 0x472000 thru 0x47c9dd (0xa9de bytes) 000499[1] 1777045455.612956 trunc from 0x8af000 to 0x3dedb3 (0x4d024c bytes) All operations completed A-OK! PASS 16a (215s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 11:44:36 (1777045476) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.5604 s, 5.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.96867 s, 4.3 MB/s Chance of close/open is 1 in 50 Seed set to 5487 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x4655ab 000001[1] 1777045491.369978 trunc from 00000000 to 0x4655aa (0x4655ab bytes) 000005[0] 1777045491.798062 mapread 0x2a8b98 thru 0x2a9981 (0xdea bytes) 000006[0] 1777045492.019852 write 0x6e85aa thru 0x6e9856 (0x12ad bytes) 000009[1] 1777045492.205115 write 0x65e7b6 thru 0x65f30d (0xb58 bytes) 000011[0] 1777045492.564538 trunc from 0x6e9857 to 0x3062c6 (0x3e3590 bytes) 000013[0] 1777045492.897987 mapread 0x2308fc thru 0x230d22 (0x427 bytes) truncating to largest ever: 0x7acc88 000015[1] 1777045493.217226 trunc from 0x3062c7 to 0x7acc87 (0x4a69c1 bytes) 000016[1] 1777045493.575787 mapwrite 0x7b1106 thru 0x7b2245 (0x1140 bytes) 000018[1] 1777045494.327175 trunc from 0x7b2246 to 0x784f4b (0x2d2fa bytes) 000020[1] 1777045494.765589 read 0x12b104 thru 0x12ba6e (0x96b bytes) 000022[0] 1777045494.998164 write 0x93be2a thru 0x93c34c (0x523 bytes) 000027[1] 1777045495.531005 mapread 0x6df4e0 thru 0x6dfefa (0xa1b bytes) 000029[0] 1777045495.731063 read 0x4eab12 thru 0x4ec724 (0x1c13 bytes) 000030[1] 1777045495.820689 mapwrite 0x5db3d0 thru 0x5dc7e9 (0x141a bytes) 000037[0] 1777045496.957628 write 0x084fa0 thru 0x086d66 (0x1dc7 bytes) 000038[0] 1777045497.133614 read 0x0ab1bd thru 0x0ac163 (0xfa7 bytes) 000041[0] 1777045497.484269 mapwrite 0x0029d6 thru 0x003513 (0xb3e bytes) 000042[1] 1777045498.152722 write 0x644964 thru 0x644a45 (0xe2 bytes) 000044[0] 1777045498.400103 mapread 0x59887a thru 0x59a060 (0x17e7 bytes) 000045[1] 1777045498.751502 mapwrite 0x401701 thru 0x402b99 (0x1499 bytes) 000046[1] 1777045499.409609 write 0x4a33dd thru 0x4a3682 (0x2a6 bytes) truncating to largest ever: 0x8e7bb3 000051[0] 1777045499.989087 trunc from 0x93c34d to 0x8e7bb2 (0x5479a bytes) 000054[0] 1777045500.530841 trunc from 0x8e7bb3 to 0x3abe6c (0x53bd46 bytes) 000056[1] 1777045500.813153 mapwrite 0x4b02c6 thru 0x4b0d01 (0xa3c bytes) 000057[1] 1777045501.604115 mapread 0x3741bc thru 0x374878 (0x6bd bytes) 000058[1] 1777045501.750212 mapread 0x0bb33e thru 0x0bca1e (0x16e1 bytes) 000060[1] 1777045502.033261 read 0x33844b thru 0x33a2d4 (0x1e8a bytes) 000062[1] 1777045502.257110 write 0x95b236 thru 0x95b385 (0x150 bytes) 000067[0] 1777045502.771976 trunc from 0x95b386 to 0x711e66 (0x24951f bytes) 000069[0] 1777045503.174472 write 0x65c2c4 thru 0x65df90 (0x1ccd bytes) 000070[0] 1777045503.433253 write 0x5a9549 thru 0x5ab2b2 (0x1d6a bytes) 000072[1] 1777045503.662138 read 0x40f50c thru 0x4111a4 (0x1c99 bytes) 000074[0] 1777045503.867048 mapread 0x1b44c5 thru 0x1b617b (0x1cb7 bytes) 000079[1] 1777045504.270258 mapwrite 0x7ff6f4 thru 0x7ffd00 (0x60d bytes) 000080[0] 1777045504.959404 mapread 0x71a5e4 thru 0x71a89a (0x2b7 bytes) 000082[0] 1777045505.262549 trunc from 0x7ffd01 to 0x152e26 (0x6aceda bytes) 000087[0] 1777045506.247621 mapread 0x092fdc thru 0x09324a (0x26f bytes) 000088[0] 1777045506.453252 mapwrite 0x24931d thru 0x249442 (0x126 bytes) 000089[1] 1777045506.964414 write 0x303fe9 thru 0x304ad9 (0xaf1 bytes) 000091[0] 1777045507.327565 read 0x1c35a9 thru 0x1c35f8 (0x50 bytes) 000092[1] 1777045507.940330 trunc from 0x304ada to 0x310cbc (0xc1e3 bytes) 000093[0] 1777045508.273690 read 0x1316be thru 0x1325da (0xf1d bytes) 000097[1] 1777045509.201524 write 0x0246c7 thru 0x0259ec (0x1326 bytes) 000105[1] 1777045509.928410 write 0x12b1af thru 0x12be9d (0xcef bytes) 000106[0] 1777045510.105380 trunc from 0x310cbd to 0x89b3ea (0x58a72e bytes) 000108[1] 1777045510.658777 write 0x3c9472 thru 0x3ca11a (0xca9 bytes) 000109[0] 1777045510.920618 read 0x7881a2 thru 0x78996f (0x17ce bytes) 000110[0] 1777045511.049216 write 0x3c3d6f thru 0x3c5383 (0x1615 bytes) truncating to largest ever: 0x8fb239 000111[1] 1777045511.330312 trunc from 0x89b3eb to 0x8fb238 (0x5fe4e bytes) 000112[1] 1777045511.522110 read 0x53dddb thru 0x53e98a (0xbb0 bytes) 000113[1] 1777045511.566136 write 0x9d672d thru 0x9d8116 (0x19ea bytes) 000114[1] 1777045511.592874 mapread 0x907204 thru 0x907264 (0x61 bytes) 000115[0] 1777045511.760265 read 0x2a8c53 thru 0x2aa73a (0x1ae8 bytes) 000121[1] 1777045512.170654 trunc from 0x9d8117 to 0x6f860a (0x2dfb0c bytes) 000122[0] 1777045512.392447 mapread 0x67a08d thru 0x67a53d (0x4b1 bytes) 000123[0] 1777045512.544300 mapread 0x3c8849 thru 0x3c8a26 (0x1de bytes) 000126[0] 1777045512.714077 mapread 0x117464 thru 0x118a33 (0x15d0 bytes) 000128[0] 1777045512.819114 trunc from 0x6f860b to 0x47cf5b (0x27b6af bytes) 000129[0] 1777045513.190379 mapwrite 0x7a3098 thru 0x7a4a9f (0x1a08 bytes) 000131[1] 1777045513.853046 trunc from 0x7a4aa0 to 0x3dc29c (0x3c8803 bytes) 000132[1] 1777045514.174302 mapwrite 0x449adc thru 0x44a3fc (0x921 bytes) 000134[0] 1777045514.725747 write 0x00773e thru 0x009194 (0x1a57 bytes) 000136[0] 1777045515.139345 mapread 0x3ea6dc thru 0x3ec1d7 (0x1afc bytes) 000137[1] 1777045515.479721 write 0x3de99c thru 0x3e058f (0x1bf4 bytes) 000140[0] 1777045515.880237 mapread 0x2d059e thru 0x2d1499 (0xefc bytes) 000142[0] 1777045516.168174 write 0x6af243 thru 0x6b07db (0x1599 bytes) 000143[0] 1777045516.239007 read 0x1ff962 thru 0x1ffee7 (0x586 bytes) 000144[0] 1777045516.436954 mapread 0x63f15f thru 0x63fc03 (0xaa5 bytes) 000145[0] 1777045516.550664 trunc from 0x6b07dc to 0x26110d (0x44f6ce bytes) 000152[0] 1777045517.057042 read 0x0b4e52 thru 0x0b6c07 (0x1db6 bytes) 000154[0] 1777045517.619084 mapwrite 0x135086 thru 0x136d6a (0x1ce5 bytes) 000157[1] 1777045518.149935 mapread 0x0c14da thru 0x0c15d5 (0xfc bytes) 000158[1] 1777045518.379610 read 0x21962b thru 0x21b581 (0x1f57 bytes) 000161[1] 1777045518.897343 mapwrite 0x0a8bf1 thru 0x0aa0c5 (0x14d5 bytes) 000163[1] 1777045519.426093 write 0x1bf9b8 thru 0x1c0f87 (0x15d0 bytes) 000164[1] 1777045519.670807 mapread 0x0c5103 thru 0x0c5496 (0x394 bytes) 000167[1] 1777045520.055819 mapread 0x17b94c thru 0x17c1e1 (0x896 bytes) 000169[1] 1777045520.317076 trunc from 0x26110e to 0x8c0902 (0x65f7f5 bytes) 000170[1] 1777045520.654957 trunc from 0x8c0903 to 0x8f419e (0x3389c bytes) 000172[1] 1777045520.970250 read 0x76c5eb thru 0x76dcdb (0x16f1 bytes) 000174[1] 1777045521.156639 write 0x7a3796 thru 0x7a42da (0xb45 bytes) 000175[0] 1777045521.202721 trunc from 0x8f419f to 0x5b9718 (0x33aa86 bytes) 000176[1] 1777045521.528708 mapread 0x21424a thru 0x214597 (0x34e bytes) 000177[1] 1777045521.876503 mapwrite 0x1a0332 thru 0x1a0c30 (0x8ff bytes) 000178[1] 1777045522.442433 read 0x500b28 thru 0x5022d7 (0x17b0 bytes) 000182[0] 1777045522.811476 mapwrite 0x19ba0a thru 0x19d157 (0x174e bytes) 000186[0] 1777045523.933862 trunc from 0x5b9719 to 0x87a1b0 (0x2c0a98 bytes) 000187[0] 1777045524.094551 read 0x5ccfbc thru 0x5ce2d8 (0x131d bytes) 000188[1] 1777045524.134169 write 0x043d8e thru 0x0451c2 (0x1435 bytes) 000189[1] 1777045524.428626 trunc from 0x87a1b1 to 0x7aed7c (0xcb434 bytes) 000191[1] 1777045524.784828 read 0x05890b thru 0x05a3d1 (0x1ac7 bytes) 000192[0] 1777045524.830144 mapread 0x05bdb3 thru 0x05d1b9 (0x1407 bytes) 000193[1] 1777045525.035987 read 0x6f5153 thru 0x6f5716 (0x5c4 bytes) truncating to largest ever: 0x931345 000194[0] 1777045525.161586 trunc from 0x7aed7d to 0x931344 (0x1825c8 bytes) 000195[1] 1777045525.546104 mapwrite 0x6fc6d2 thru 0x6fc70b (0x3a bytes) 000196[1] 1777045526.018218 read 0x7cf899 thru 0x7d0fba (0x1722 bytes) 000197[1] 1777045526.144786 write 0x24f6ad thru 0x24fa0d (0x361 bytes) 000198[1] 1777045526.256812 trunc from 0x931345 to 0x768b57 (0x1c87ed bytes) 000199[0] 1777045526.457959 read 0x35f090 thru 0x36036e (0x12df bytes) 000200[1] 1777045526.675400 read 0x1b4091 thru 0x1b4f79 (0xee9 bytes) 000202[0] 1777045526.906915 trunc from 0x768b58 to 0x4fcce6 (0x26be71 bytes) 000205[1] 1777045527.376912 read 0x016a2c thru 0x0178c7 (0xe9c bytes) 000215[0] 1777045527.691803 write 0x47367e thru 0x4751bf (0x1b42 bytes) 000217[0] 1777045527.801717 mapread 0x45668f thru 0x456757 (0xc9 bytes) 000220[1] 1777045527.980486 write 0x336e28 thru 0x337f49 (0x1122 bytes) 000221[1] 1777045528.157749 mapwrite 0x8816f7 thru 0x88238c (0xc96 bytes) 000223[1] 1777045528.801650 trunc from 0x88238d to 0x88ec9d (0xc911 bytes) 000227[1] 1777045529.245434 read 0x52e974 thru 0x52eb05 (0x192 bytes) 000229[1] 1777045529.463898 write 0x46c025 thru 0x46cd5a (0xd36 bytes) 000231[0] 1777045529.583589 write 0x03fec6 thru 0x041232 (0x136d bytes) 000232[1] 1777045529.752721 trunc from 0x88ec9e to 0x03d82b (0x851472 bytes) 000233[1] 1777045529.941051 trunc from 0x03d82c to 0x5215d8 (0x4e3dad bytes) 000235[1] 1777045530.228024 write 0x9285c5 thru 0x929c7b (0x16b7 bytes) 000236[0] 1777045530.255107 mapwrite 0x7b87e5 thru 0x7b8a73 (0x28f bytes) 000237[0] 1777045530.970676 mapwrite 0x9f5553 thru 0x9f7306 (0x1db4 bytes) 000240[1] 1777045531.678143 trunc from 0x9f7307 to 0x2f9715 (0x6fdbf1 bytes) 000241[1] 1777045531.968488 mapread 0x155e22 thru 0x157b8f (0x1d6e bytes) 000245[0] 1777045532.368032 mapread 0x07523e thru 0x076044 (0xe07 bytes) 000246[1] 1777045532.630523 read 0x1d208f thru 0x1d2762 (0x6d4 bytes) 000247[0] 1777045532.974124 mapwrite 0x3ba011 thru 0x3baf56 (0xf46 bytes) 000250[0] 1777045533.880320 write 0x4549ee thru 0x456370 (0x1983 bytes) 000254[0] 1777045534.149405 write 0x088372 thru 0x089dc8 (0x1a57 bytes) 000263[0] 1777045534.785772 mapwrite 0x2a1e8c thru 0x2a3e81 (0x1ff6 bytes) 000265[1] 1777045535.137780 write 0x397602 thru 0x3983b8 (0xdb7 bytes) 000266[1] 1777045535.659828 write 0x988895 thru 0x98a517 (0x1c83 bytes) 000267[0] 1777045535.788343 mapread 0x630220 thru 0x63105c (0xe3d bytes) 000269[1] 1777045536.254096 read 0x73e071 thru 0x73ed3b (0xccb bytes) 000271[0] 1777045536.424466 mapwrite 0x1dc08e thru 0x1dd765 (0x16d8 bytes) 000274[0] 1777045537.140771 write 0x3b270b thru 0x3b404e (0x1944 bytes) 000278[0] 1777045537.387247 mapwrite 0x4e53af thru 0x4e69c3 (0x1615 bytes) 000279[0] 1777045537.951509 mapread 0x7a1c70 thru 0x7a3b0f (0x1ea0 bytes) 000280[1] 1777045538.023467 read 0x436be3 thru 0x4372ca (0x6e8 bytes) 000282[1] 1777045538.345706 read 0x39eb29 thru 0x39f1c9 (0x6a1 bytes) 000283[1] 1777045538.464531 trunc from 0x98a518 to 0x1c52c8 (0x7c524f bytes) 000284[1] 1777045538.733199 read 0x13b7c3 thru 0x13ceb1 (0x16ef bytes) 000285[0] 1777045539.010519 mapread 0x07a03c thru 0x07abba (0xb7f bytes) 000286[1] 1777045539.311339 write 0x03c30b thru 0x03df40 (0x1c36 bytes) 000289[0] 1777045539.427238 read 0x12b0ea thru 0x12ccce (0x1be5 bytes) 000293[0] 1777045539.826166 read 0x05eee2 thru 0x0601f2 (0x1311 bytes) 000294[1] 1777045539.827470 write 0x1bc64b thru 0x1bdaaf (0x1465 bytes) 000297[1] 1777045540.274171 mapwrite 0x028f8d thru 0x02a3f2 (0x1466 bytes) 000308[0] 1777045541.377118 mapwrite 0x7b7a8b thru 0x7b9355 (0x18cb bytes) 000309[0] 1777045542.380850 mapwrite 0x166aca thru 0x168749 (0x1c80 bytes) 000310[1] 1777045542.625659 write 0x9b88ba thru 0x9b91c9 (0x910 bytes) 000311[0] 1777045542.751690 write 0x560aec thru 0x561206 (0x71b bytes) 000314[1] 1777045542.967508 mapwrite 0x2cee4a thru 0x2d0e00 (0x1fb7 bytes) 000316[1] 1777045543.778063 mapread 0x6ccc65 thru 0x6ce2c2 (0x165e bytes) 000319[0] 1777045544.526964 write 0x190d6c thru 0x191973 (0xc08 bytes) 000323[1] 1777045545.011340 mapwrite 0x42bd4e thru 0x42c8b4 (0xb67 bytes) 000324[0] 1777045545.508137 read 0x257f9d thru 0x258bac (0xc10 bytes) truncating to largest ever: 0x99cc24 000325[0] 1777045545.603533 trunc from 0x9b91ca to 0x99cc23 (0x1c5a6 bytes) 000326[0] 1777045545.844815 mapread 0x499ee4 thru 0x49b4d1 (0x15ee bytes) 000327[1] 1777045545.959235 mapread 0x11f9ba thru 0x1201e6 (0x82d bytes) 000330[0] 1777045546.217617 write 0x8773c4 thru 0x87863a (0x1277 bytes) 000331[1] 1777045546.340824 write 0x3d0ac5 thru 0x3d12fc (0x838 bytes) 000332[1] 1777045546.543579 write 0x485360 thru 0x485c9d (0x93e bytes) 000334[1] 1777045546.722843 mapwrite 0x0f5464 thru 0x0f700f (0x1bac bytes) 000337[1] 1777045547.091666 read 0x676a03 thru 0x6777b2 (0xdb0 bytes) 000343[1] 1777045547.671362 mapread 0x384c22 thru 0x385016 (0x3f5 bytes) 000345[1] 1777045547.735448 read 0x6f80a0 thru 0x6f8db6 (0xd17 bytes) 000347[0] 1777045547.928558 write 0x46dd84 thru 0x46fa9d (0x1d1a bytes) 000348[1] 1777045548.239859 mapwrite 0x7829b5 thru 0x783922 (0xf6e bytes) 000353[1] 1777045548.742932 mapread 0x94b4ef thru 0x94c9f1 (0x1503 bytes) 000355[1] 1777045548.787658 mapread 0x3c310a thru 0x3c4286 (0x117d bytes) 000356[0] 1777045548.917060 read 0x9020d1 thru 0x902f93 (0xec3 bytes) 000358[0] 1777045549.070828 mapwrite 0x5f7550 thru 0x5f7c26 (0x6d7 bytes) 000359[0] 1777045549.367817 read 0x5dee80 thru 0x5df8ba (0xa3b bytes) 000360[1] 1777045549.494430 trunc from 0x99cc24 to 0x59a395 (0x40288e bytes) 000361[1] 1777045549.865183 write 0x7b8d36 thru 0x7b90dc (0x3a7 bytes) 000362[0] 1777045549.921613 mapwrite 0x826b8a thru 0x826e04 (0x27b bytes) 000363[0] 1777045550.448425 mapwrite 0x88a54e thru 0x88a5f8 (0xab bytes) 000364[0] 1777045550.846987 mapread 0x7c3806 thru 0x7c4016 (0x811 bytes) 000366[1] 1777045551.187538 mapwrite 0x2592da thru 0x25a98c (0x16b3 bytes) 000367[1] 1777045551.843441 read 0x04f033 thru 0x0509ec (0x19ba bytes) 000368[0] 1777045551.964581 mapread 0x74a58a thru 0x74b517 (0xf8e bytes) 000370[0] 1777045552.350172 mapwrite 0x97459e thru 0x975224 (0xc87 bytes) 000372[0] 1777045553.112382 read 0x68f3df thru 0x68f95d (0x57f bytes) 000373[0] 1777045553.285569 read 0x00d8bc thru 0x00f25a (0x199f bytes) 000375[0] 1777045553.500261 mapread 0x7eb29e thru 0x7ec114 (0xe77 bytes) 000380[1] 1777045553.576482 mapwrite 0x842e4c thru 0x844653 (0x1808 bytes) 000382[0] 1777045554.145080 mapwrite 0x1f3471 thru 0x1f404f (0xbdf bytes) 000383[1] 1777045554.591069 trunc from 0x975225 to 0x2061d1 (0x76f053 bytes) 000390[1] 1777045555.471715 mapread 0x19c718 thru 0x19c87b (0x164 bytes) 000391[1] 1777045555.674519 trunc from 0x2061d2 to 0x0e1b2a (0x1246a7 bytes) 000392[1] 1777045555.794185 trunc from 0x0e1b2b to 0x79c01a (0x6ba4f0 bytes) 000393[1] 1777045555.977544 trunc from 0x79c01b to 0x217995 (0x584685 bytes) 000396[0] 1777045556.386682 trunc from 0x217996 to 0x3bf0c9 (0x1a7734 bytes) 000398[1] 1777045556.722771 write 0x48a704 thru 0x48a863 (0x160 bytes) 000403[1] 1777045557.361351 mapwrite 0x01f551 thru 0x01ff74 (0xa24 bytes) 000404[0] 1777045557.874172 read 0x093163 thru 0x094a83 (0x1921 bytes) 000406[0] 1777045557.988513 mapread 0x16258f thru 0x1625ec (0x5e bytes) 000407[0] 1777045558.188646 trunc from 0x48a864 to 0x2864c5 (0x20439e bytes) 000409[1] 1777045558.578072 trunc from 0x2864c6 to 0x17ca49 (0x109a7c bytes) 000411[0] 1777045558.985630 write 0x8d1a1b thru 0x8d21e3 (0x7c9 bytes) 000416[1] 1777045559.483751 mapread 0x7f644a thru 0x7f70c7 (0xc7e bytes) 000417[1] 1777045559.704758 mapread 0x21d474 thru 0x21d61a (0x1a7 bytes) 000418[0] 1777045559.983723 trunc from 0x8d21e4 to 0x3c2ba4 (0x50f63f bytes) 000419[0] 1777045560.188698 write 0x951055 thru 0x951b23 (0xacf bytes) 000420[1] 1777045560.199196 read 0x60a6de thru 0x60b1f2 (0xb15 bytes) truncating to largest ever: 0x9af485 000421[1] 1777045560.411619 trunc from 0x951b24 to 0x9af484 (0x5d961 bytes) 000425[1] 1777045560.706378 write 0x5b74cb thru 0x5b9312 (0x1e48 bytes) 000426[0] 1777045560.881537 write 0x451252 thru 0x4519fb (0x7aa bytes) 000427[1] 1777045561.025523 write 0x82c11e thru 0x82c5a4 (0x487 bytes) 000429[1] 1777045561.214007 trunc from 0x9af485 to 0x872535 (0x13cf4f bytes) 000431[1] 1777045561.594154 write 0x720a2d thru 0x7216bb (0xc8f bytes) 000432[0] 1777045561.897509 read 0x625359 thru 0x6264bc (0x1164 bytes) 000437[1] 1777045562.325819 trunc from 0x872536 to 0x4ac18e (0x3c63a7 bytes) 000443[0] 1777045562.930421 mapwrite 0x99a49c thru 0x99bd13 (0x1878 bytes) 000444[0] 1777045563.612053 mapread 0x70f89c thru 0x7104d3 (0xc38 bytes) 000445[1] 1777045563.670658 mapwrite 0x3d501f thru 0x3d5800 (0x7e2 bytes) 000447[1] 1777045564.189260 mapwrite 0x5ec523 thru 0x5edbf3 (0x16d1 bytes) 000448[1] 1777045564.612950 mapwrite 0x51624d thru 0x51662d (0x3e1 bytes) 000449[0] 1777045564.903606 trunc from 0x99bd14 to 0x43ef09 (0x55ce0a bytes) 000451[0] 1777045565.342755 trunc from 0x43ef0a to 0x4a98c0 (0x6a9b7 bytes) 000452[1] 1777045565.632346 trunc from 0x4a98c1 to 0x07e3aa (0x42b516 bytes) 000453[1] 1777045565.959952 write 0x178869 thru 0x178a97 (0x22f bytes) 000454[0] 1777045566.122600 mapread 0x05ca19 thru 0x05d841 (0xe29 bytes) 000462[0] 1777045567.003219 mapread 0x0c3300 thru 0x0c3dd4 (0xad5 bytes) 000463[0] 1777045567.094897 trunc from 0x178a98 to 0x1aa35c (0x318c5 bytes) 000464[0] 1777045567.365630 write 0x34d5e0 thru 0x34d9e3 (0x404 bytes) 000465[1] 1777045567.474250 trunc from 0x34d9e4 to 0x81115b (0x4c3778 bytes) 000466[0] 1777045567.770055 mapread 0x6fbb84 thru 0x6fcbf6 (0x1073 bytes) 000467[1] 1777045568.093707 mapwrite 0x529927 thru 0x52b7fc (0x1ed6 bytes) 000469[0] 1777045568.738066 read 0x257aae thru 0x258c57 (0x11aa bytes) 000470[0] 1777045568.891416 trunc from 0x81115c to 0x93567f (0x124524 bytes) 000472[0] 1777045569.229054 mapwrite 0x65e436 thru 0x65fd85 (0x1950 bytes) 000473[0] 1777045569.784640 write 0x809a9b thru 0x80b29f (0x1805 bytes) 000474[1] 1777045569.828852 read 0x898f67 thru 0x89a407 (0x14a1 bytes) 000476[1] 1777045570.121401 mapread 0x0c333b thru 0x0c3515 (0x1db bytes) 000478[1] 1777045570.245559 read 0x066a3d thru 0x067589 (0xb4d bytes) 000479[1] 1777045570.367124 trunc from 0x935680 to 0x900ed8 (0x347a7 bytes) 000481[0] 1777045570.773129 read 0x387f8b thru 0x389488 (0x14fe bytes) 000486[0] 1777045571.246583 mapwrite 0x10e843 thru 0x11066c (0x1e2a bytes) 000490[1] 1777045572.481038 mapwrite 0x046633 thru 0x04789b (0x1269 bytes) 000491[1] 1777045573.342652 mapwrite 0x5c783d thru 0x5c7d63 (0x527 bytes) 000495[0] 1777045573.970418 mapwrite 0x2d858f thru 0x2d8f7a (0x9ec bytes) 000498[0] 1777045575.006736 read 0x6cfa73 thru 0x6d0f1e (0x14ac bytes) 000500[1] 1777045575.250922 trunc from 0x900ed9 to 0x97b4d4 (0x7a5fc bytes) All operations completed A-OK! PASS 16b (121s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 11:46:38 (1777045598) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (15s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 11:46:53 (1777045613) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.58606 s, 5.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.61165 s, 5.2 MB/s ldlm.namespaces.MGC192.168.201.159@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff97ee87d59000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff97ee89f26000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff97ee89f26000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff97ee89f26000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 52.9374 s, 2.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4444 s, 8.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.84848 s, 11.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7181 s, 5.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 27.8393 s, 3.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1646 s, 8.0 MB/s PASS 16d (178s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 11:49:51 (1777045791) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.18754 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352728 s, 3.0 MB/s PASS 16e (20s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 11:50:11 (1777045811) TIMEOUT PASS 16f (43s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 11:50:54 (1777045854) TIMEOUT PASS 16g (43s) == sanityn test 16h: mmap read after truncate file ======= 11:51:37 (1777045897) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (21s) == sanityn test 16i: read after truncate file ============ 11:51:59 (1777045919) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (22s) == sanityn test 16j: race dio with buffered i/o ========== 11:52:20 (1777045940) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 11.0032 s, 2.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.53138 s, 4.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 11.7543 s, 2.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.8537 s, 4.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 8.90328 s, 2.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 9.26298 s, 2.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 12.3651 s, 2.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.27542 s, 6.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.60662 s, 4.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 12.5116 s, 2.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 13.53 s, 1.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.25856 s, 3.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.92392 s, 3.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 10.5498 s, 2.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 10.8811 s, 2.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 11.1449 s, 2.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 12.7968 s, 2.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 14.8934 s, 1.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 15.1592 s, 1.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (154s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 11:54:54 (1777046094) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Apr 24 11:57:12 EDT 2026 + fsx 2 done: rc=0 Fri Apr 24 11:57:12 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Fri Apr 24 11:57:22 EDT 2026 + fsx 4 done: rc=0 Fri Apr 24 11:57:22 EDT 2026 + fsx 5 done: rc=0 Fri Apr 24 11:57:22 EDT 2026 PASS 16k (184s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 11:57:59 (1777046279) fail_loc=0x8000030a PASS 17 (28s) == sanityn test 18: mmap sanity check =========================================================================================== 11:58:28 (1777046308) mmap test1: basic mmap operation (PASS, 0.47793s) mmap test2: MAP_PRIVATE not write back (PASS, 0.35304s) mmap test3: concurrent mmap ops on two nodes (PASS, 16.803s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 5.9203s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.61329s) mmap test6: check mmap write/read content on two nodes (PASS, 0.77217s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 6.1489s) mmap test11: multi-thread shared mmap (PASS, 7.7841s) mmap test10: mtime not change for readonly mmap access (PASS, 1.3543s) PASS 18 (65s) == sanityn test 19: test concurrent uncached read races ========================================================================= 11:59:33 (1777046373) oleg159-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (13s) == sanityn test 20: test extra readahead page left in cache ============================================================== 11:59:46 (1777046386) PASS 20 (23s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 12:00:09 (1777046409) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (22s) == sanityn test 23: others should see updated atime while another read============================================================== 12:00:31 (1777046431) now is 1777046440 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 new atime is 1777046501 PASS 23 (90s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 12:02:01 (1777046521) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (26s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 12:02:28 (1777046548) PASS 24b (21s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 12:02:49 (1777046569) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (25s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 12:03:14 (1777046594) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (11s) == sanityn test 26a: allow mtime to get older ============ 12:03:26 (1777046606) utime: good mknod times 1777046609 <= 1777046610 <= 1777046610 for /mnt/lustre/f26a utime: good mknod times 1777046609 <= 1777046610 <= 1777046610 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (24s) == sanityn test 26b: sync mtime between ost and mds ====== 12:03:49 (1777046629) PASS 26b (24s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 12:04:14 (1777046654) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-24 12:04:19.000000000 -0400 Birth: 2026-04-24 12:04:20.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-24 12:04:19.000000000 -0400 Birth: 2026-04-24 12:04:20.000000000 -0400 PASS 26c (23s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 12:04:41 (1777046681) 0 PASS 30 (24s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 12:05:05 (1777046705) fail_loc=0x314 PASS 31a (21s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 12:05:26 (1777046726) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg159-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.159@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (49s) == sanityn test 31r: open-rename(replace) race =========== 12:06:16 (1777046776) -rw-r--r-- 1 root root 0 Apr 24 12:06 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (25s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 12:06:41 (1777046801) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43305 Terminated tail -f $tfile PASS 31s (21s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 12:07:02 (1777046822) PASS 31t (20s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 12:07:28 (1777046848) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (11s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 12:07:39 (1777046859) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (10s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 12:07:49 (1777046869) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (10s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 12:08:00 (1777046880) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (10s) == sanityn test 34: no lock timeout under IO ============= 12:08:10 (1777046890) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ee87d59000.ost_server_uuid in IDLE state after 0 sec PASS 34 (237s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 12:12:08 (1777047128) fail_loc=0x80000317 Race attempt 0 Wait for 49386 49405 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1746: 49386 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.159@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.201.159@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 203 (time 1777047156.83 total 10.01 last 20.27) - open/close 401 (time 1777047166.87 total 20.06 last 19.72) - open/close 610 (time 1777047176.89 total 30.07 last 20.86) - open/close 842 (time 1777047186.91 total 40.09 last 23.16) - open/close 1076 (time 1777047196.91 total 50.10 last 23.38) - open/close 1296 (time 1777047206.97 total 60.15 last 21.87) - open/close 1502 (time 1777047217.00 total 70.18 last 20.54) - open/close 1696 (time 1777047227.03 total 80.21 last 19.34) - open/close 1913 (time 1777047237.06 total 90.24 last 21.63) - open/close 2107 (time 1777047247.10 total 100.28 last 19.33) - open/close 2307 (time 1777047257.14 total 110.32 last 19.91) - open/close 2498 (time 1777047267.14 total 120.32 last 19.10) - open/close 2688 (time 1777047277.17 total 130.35 last 18.95) - open/close 2904 (time 1777047287.21 total 140.39 last 21.52) - open/close 3137 (time 1777047297.21 total 150.40 last 23.28) - open/close 3372 (time 1777047307.22 total 160.40 last 23.49) - open/close 3611 (time 1777047317.22 total 170.41 last 23.89) - open/close 3834 (time 1777047327.28 total 180.47 last 22.17) total: 4000 open/close in 187.19 seconds: 21.37 ops/second PASS 35 (207s) == sanityn test 36: handle ESTALE/open-unlink correctly == 12:15:34 (1777047334) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg159-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.61987 s, 20.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.20126 s, 23.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95487 s, 26.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.07734 s, 25.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69125 s, 31.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.96114 s, 26.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.00467 s, 26.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.15303 s, 24.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.83841 s, 28.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7429120) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78233 s, 29.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08638 s, 25.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7439360) after(7496704) PASS 36 (484s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 12:23:38 (1777047818) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 - create 504 (time 1777047836.98 total 10.02 last 50.28) - create 966 (time 1777047847.00 total 20.03 last 46.15) - create 1341 (time 1777047857.04 total 30.08 last 37.32) - create 1720 (time 1777047867.05 total 40.09 last 37.86) - create 2096 (time 1777047877.07 total 50.11 last 37.53) - create 2516 (time 1777047887.08 total 60.12 last 41.97) - create 3148 (time 1777047897.09 total 70.13 last 63.15) - create 3740 (time 1777047907.10 total 80.14 last 59.15) - create 4343 (time 1777047917.11 total 90.14 last 60.25) - create 4911 (time 1777047927.13 total 100.17 last 56.63) - create 5393 (time 1777047937.15 total 110.19 last 48.10) - create 5841 (time 1777047947.17 total 120.21 last 44.71) - create 6397 (time 1777047957.19 total 130.23 last 55.52) - create 6955 (time 1777047967.20 total 140.24 last 55.75) - create 7519 (time 1777047977.20 total 150.24 last 56.37) - create 8112 (time 1777047987.21 total 160.25 last 59.28) - create 8731 (time 1777047997.21 total 170.25 last 61.88) - create 9395 (time 1777048007.22 total 180.26 last 66.34) - create 9879 (time 1777048017.22 total 190.26 last 48.38) total: 10000 create in 192.08 seconds: 52.06 ops/second PASS 37 (216s) == sanityn test 39a: file mtime does not change after rename ========================================================== 12:27:14 (1777048034) repeat after cancel_lru_locks PASS 39a (15s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 12:27:29 (1777048049) repeat after cancel_lru_locks PASS 39b (13s) == sanityn test 39c: check truncate mtime update ================================================================================ 12:27:42 (1777048062) repeat after cancel_lru_locks PASS 39c (14s) == sanityn test 39d: sync write should update mtime ====== 12:27:57 (1777048077) fail_loc=0x411 fail_loc=0 PASS 39d (9s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 12:28:08 (1777048088) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (25s) == sanityn test 40c: pdirops: link and others ======================================================================== 12:28:33 (1777048113) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (21s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 12:28:54 (1777048134) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (23s) == sanityn test 40e: pdirops: rename and others ======================================================================== 12:29:17 (1777048157) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (22s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 12:29:39 (1777048179) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (18s) == sanityn test 41b: pdirops: create vs create ======================================================================== 12:29:57 (1777048197) open: File exists Conflict PASS 41b (18s) == sanityn test 41c: pdirops: create vs link ======================================================================== 12:30:15 (1777048215) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (18s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 12:30:34 (1777048234) Conflict PASS 41d (17s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 12:30:51 (1777048251) Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 12:31:08 (1777048268) Conflict PASS 41f (18s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 12:31:26 (1777048286) Conflict PASS 41g (18s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 12:31:44 (1777048304) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (22s) == sanityn test 41i: reint_open: create vs create ======== 12:32:06 (1777048326) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1177s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 12:51:43 (1777049503) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 12:51:56 (1777049516) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 12:52:08 (1777049528) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (12s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 12:52:19 (1777049539) Conflict PASS 42d (11s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 12:52:31 (1777049551) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (11s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 12:52:43 (1777049563) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 12:52:54 (1777049574) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 12:53:06 (1777049586) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 12:53:19 (1777049599) PASS 43a (57s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 12:54:16 (1777049656) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 12:54:28 (1777049668) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 12:54:40 (1777049680) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 12:54:52 (1777049692) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 12:55:04 (1777049704) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 12:55:16 (1777049716) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 12:55:29 (1777049729) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 12:55:42 (1777049742) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 12:55:46 (1777049746) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (113s) == sanityn test 43k: unlink vs create ==================== 12:57:38 (1777049858) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1076s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 13:15:34 (1777050934) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Apr 24 13:15:41 EDT 2026 PASS 44a (11s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:15:45 (1777050945) open: File exists Conflict PASS 44b (11s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:15:57 (1777050957) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (11s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:16:08 (1777050968) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:16:19 (1777050979) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:16:31 (1777050991) Conflict PASS 44f (12s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:16:43 (1777051003) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:16:55 (1777051015) Conflict PASS 44h (12s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:17:07 (1777051027) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:17:09 (1777051029) PASS 45a (92s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:18:41 (1777051121) Conflict PASS 45b (9s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 13:18:50 (1777051130) Conflict PASS 45c (8s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 13:18:59 (1777051139) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (10s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 13:19:09 (1777051149) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 13:19:19 (1777051159) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 13:19:29 (1777051169) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (10s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 13:19:39 (1777051179) Conflict PASS 45h (10s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 13:19:49 (1777051189) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 13:19:52 (1777051192) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (930s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 13:35:22 (1777052122) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 13:35:33 (1777052133) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 13:35:44 (1777052144) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (11s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 13:35:55 (1777052155) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 13:36:06 (1777052166) Conflict PASS 46e (9s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 13:36:15 (1777052175) Conflict PASS 46f (9s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 13:36:24 (1777052184) Conflict PASS 46g (10s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 13:36:34 (1777052194) Conflict PASS 46h (9s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 13:36:44 (1777052204) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 13:36:46 (1777052206) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 13:36:48 (1777052208) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 13:36:51 (1777052211) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 13:36:54 (1777052214) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 13:36:56 (1777052216) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 13:36:58 (1777052218) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 13:37:00 (1777052220) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 13:37:03 (1777052223) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0592077 s, 173 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (10s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 13:37:13 (1777052233) 0+1 records in 0+1 records out 158 bytes copied, 0.0170261 s, 9.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217166) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 13:37:20 (1777052240) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00776276 s, 132 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295798 s, 35.4 MB/s 3145728 PASS 51b (18s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 13:37:38 (1777052258) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0114112 s, 89.7 kB/s PASS 51c (9s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 13:37:48 (1777052268) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444401 s, 23.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (6s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 13:37:53 (1777052273) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 13:38:00 (1777052280) llite.lustre-ffff97ee87d59000.xattr_cache=1 llite.lustre-ffff97ee89f26000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505008 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:01.000000000 -0400 Modify: 2026-04-24 13:38:01.000000000 -0400 Change: 2026-04-24 13:38:01.000000000 -0400 Birth: 2026-04-24 13:38:02.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:06.000000000 -0400 Modify: 2026-04-24 13:38:07.000000000 -0400 Change: 2026-04-24 13:38:07.000000000 -0400 Birth: 2026-04-24 13:38:08.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:12.000000000 -0400 Modify: 2026-04-24 13:38:12.000000000 -0400 Change: 2026-04-24 13:38:12.000000000 -0400 Birth: 2026-04-24 13:38:13.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:12.000000000 -0400 Modify: 2026-04-24 13:38:13.000000000 -0400 Change: 2026-04-24 13:38:13.000000000 -0400 Birth: 2026-04-24 13:38:13.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:18.000000000 -0400 Modify: 2026-04-24 13:38:19.000000000 -0400 Change: 2026-04-24 13:38:19.000000000 -0400 Birth: 2026-04-24 13:38:19.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 13:38:18.000000000 -0400 Modify: 2026-04-24 13:38:19.000000000 -0400 Change: 2026-04-24 13:38:19.000000000 -0400 Birth: 2026-04-24 13:38:19.000000000 -0400 PASS 54 (27s) == sanityn test 55a: rename vs unlink target dir ========= 13:38:27 (1777052307) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 13:38:37 (1777052317) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 13:38:48 (1777052328) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 13:39:05 (1777052345) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 13:39:16 (1777052356) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 13:39:19 (1777052359) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (16s) == sanityn test 55g: rename: race with trylock =========== 13:39:35 (1777052375) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (17s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 13:39:52 (1777052392) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1777052393 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.13035 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (59s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 13:40:51 (1777052451) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1777052454 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.06905 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (68s) == sanityn test 60: Verify data_version behaviour ======== 13:41:59 (1777052519) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 13:42:05 (1777052525) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 13:42:11 (1777052531) PASS 70b (6s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 13:42:17 (1777052537) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00690709 s, 9.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 13:42:20 (1777052540) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00723743 s, 9.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 13:42:23 (1777052543) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 13:42:26 (1777052546) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00763219 s, 8.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 13:42:28 (1777052548) llite.lustre-ffff97ee87d59000.xattr_cache=1 llite.lustre-ffff97ee89f26000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 13:42:33 (1777052553) llite.lustre-ffff97ee87d59000.xattr_cache=1 llite.lustre-ffff97ee89f26000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 13:42:38 (1777052558) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00793205 s, 129 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00838357 s, 122 kB/s 229951: get lock1 229950: get lock2 229951: done 229951: get lock2 229950: done 229950: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 229950: cannot get lock1: Resource deadlock avoided 229951: done 229951: exit rc=0 229950: done 229950: exit rc=0 PASS 74 (8s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 13:42:47 (1777052567) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0206991 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105303 s, 99.6 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230532 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (10s) == sanityn test 76: Verify MDT open_files listing ======== 13:42:56 (1777052576) opening files: ........ closing files: ........ PASS 76 (38s) == sanityn test 77a: check FIFO NRS policy =============== 13:43:34 (1777052614) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146515 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362429 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385329 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614244 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588734 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539636 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070948 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676879 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582389 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742505 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707861 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123135 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854543 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124845 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127051 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842026 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164821 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499225 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600155 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686124 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566652 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410873 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453014 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483141 s, 21.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716973 s, 14.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570244 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586803 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522523 s, 20.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054774 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508141 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505887 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530921 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796215 s, 13.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 13:43:41 (1777052621) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226447 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590412 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760726 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080789 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773191 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607574 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969237 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113414 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110805 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100825 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135921 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127892 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149204 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125269 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133412 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179881 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124243 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447431 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479943 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360009 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241474 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423562 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361507 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358366 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465018 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462728 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330441 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470493 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158384 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201275 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115057 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369042 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128394 s, 81.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195344 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215647 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481769 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731278 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639668 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073278 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669786 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434004 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665022 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582629 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766401 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822916 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745608 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081333 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903835 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685556 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131989 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245493 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186408 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347532 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234526 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384592 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401082 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125879 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674504 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439545 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431036 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473324 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418594 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455531 s, 23.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502427 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560681 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482059 s, 21.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (9s) == sanityn test 77c: check ORR NRS policy ================ 13:43:50 (1777052630) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264127 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485696 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766116 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815984 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783038 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999901 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077266 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632807 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905131 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929078 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795996 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609899 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564339 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161608 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183438 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191957 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231612 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325905 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645674 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066429 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053187 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799878 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670581 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639223 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06744 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705857 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682409 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723374 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572352 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659654 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973727 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715037 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709321 s, 14.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.21387 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627608 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981326 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845391 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978813 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887639 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763449 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766987 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779815 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856361 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163715 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216887 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176181 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221146 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196168 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166992 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239229 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450119 s, 23.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407908 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634358 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543097 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582359 s, 18.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597089 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652736 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130229 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672773 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699244 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748089 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726706 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756057 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764482 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101343 s, 10.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173044 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040425 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594623 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814809 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827962 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834814 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805325 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757315 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944659 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107103 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100125 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872958 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109751 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796498 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709575 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940473 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964214 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354047 s, 29.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217898 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513776 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321584 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518121 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044197 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457975 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520279 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507167 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728179 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427791 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505541 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107154 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671269 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643433 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00549083 s, 191 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (11s) == sanityn test 77d: check TRR nrs policy ================ 13:44:02 (1777052642) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.26674 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738291 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734008 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726801 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860427 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104031 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107162 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708681 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970189 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10906 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095976 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760941 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119857 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110244 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123621 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897805 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120103 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395046 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468899 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520182 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049701 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694938 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381695 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579637 s, 18.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498174 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486594 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429753 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189813 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546914 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315739 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555354 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00650858 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817549 s, 12.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248844 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598772 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912403 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111099 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153706 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168485 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160389 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156056 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128688 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134134 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120408 s, 8.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110917 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102645 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153704 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173722 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191514 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151804 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023182 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360185 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015108 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289132 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523874 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668264 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522565 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589459 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591617 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044516 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500141 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892763 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788578 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214947 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622148 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184823 s, 56.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.219808 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542114 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650675 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752238 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787292 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776721 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108078 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1037 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103396 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112157 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471096 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139905 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562906 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124722 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135432 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100525 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772898 s, 13.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612149 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341787 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331052 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488564 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672107 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632502 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665756 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587285 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777081 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526325 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601316 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507422 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817174 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065343 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629849 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668854 s, 15.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (11s) == sanityn test 77e: check TBF NID nrs policy ============ 13:44:13 (1777052653) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.59@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159378 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432236 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495869 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852932 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684482 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783608 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948985 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348455 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558614 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745782 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751881 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854352 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857337 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644783 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108756 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811221 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107284 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410013 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403861 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041433 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049903 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323555 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592478 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431137 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526469 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135239 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360692 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492498 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462197 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268974 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055457 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560286 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154756 s, 67.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.281727 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599919 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716487 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889902 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142633 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116624 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147438 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154803 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091286 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103122 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100365 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136954 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13699 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129534 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090966 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133704 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172469 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126386 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133122 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135154 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158062 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14762 s, 7.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163109 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174215 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148811 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16068 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150147 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153279 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146239 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169655 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156905 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165976 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15881 s, 6.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227193 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658699 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655606 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721955 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534159 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762602 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979936 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924514 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737389 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114681 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131726 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160251 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147312 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117589 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189682 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136142 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215333 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563229 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833316 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757568 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580984 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595697 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812473 s, 12.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593356 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753459 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634131 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660648 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747128 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754868 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801018 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759299 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833802 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804304 s, 13.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279547 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501578 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637979 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783852 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805655 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635003 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860273 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087177 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078768 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877448 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810294 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099683 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918826 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893651 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068936 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856412 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08865 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349037 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354341 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05256 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427677 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624682 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103534 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708087 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085311 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100441 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918506 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873076 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887262 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684903 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680781 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127716 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836348 s, 12.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (19s) == sanityn test 77f: check TBF JobID nrs policy ========== 13:44:32 (1777052672) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16117 s, 104 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542867 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539885 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558947 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457037 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876048 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787294 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1157 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873209 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824102 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14147 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143514 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117001 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112382 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112432 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839943 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853795 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527568 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053622 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424762 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668545 s, 15.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04706 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900124 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483073 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642632 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664383 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509072 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539842 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801455 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567228 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560999 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758871 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600292 s, 17.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154522 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431949 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435776 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553842 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936952 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958655 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118342 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107141 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867524 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112069 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908513 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884448 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956559 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111555 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918471 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129119 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480074 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590144 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488108 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488687 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532894 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053923 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551766 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538219 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055645 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060595 s, 17.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594257 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618319 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612211 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613634 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632345 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665355 s, 15.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244418 s, 68.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458437 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485208 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789407 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858738 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073333 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806052 s, 13.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932079 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106479 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977369 s, 10.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777138 s, 13.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120088 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956053 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10383 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968261 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825439 s, 12.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116832 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491075 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438624 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468648 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037957 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465503 s, 22.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569142 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590847 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542901 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575612 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590035 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932591 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607353 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302414 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664991 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337142 s, 31.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276359 s, 37.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20063 s, 83.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504081 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796233 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984235 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769851 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091449 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536055 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113656 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133323 s, 7.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120285 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108899 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854735 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160118 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118261 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127753 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141351 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97ee87d59000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff97ee87d59000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205685 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116301 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455528 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456696 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438723 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594289 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542308 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105753 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598923 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860053 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765651 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754657 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795286 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776367 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803255 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749528 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189914 s, 55.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (16s) == sanityn test 77g: Change TBF type directly ============ 13:44:48 (1777052688) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (8s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 13:44:56 (1777052696) oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg159-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg159-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg159-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg159-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 PASS 77h (9s) == sanityn test 77i: Change rank of TBF rule ============= 13:45:05 (1777052705) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (17s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 13:45:22 (1777052722) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306483 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515827 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507581 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512681 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417501 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523492 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087318 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474777 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618557 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465461 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626813 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480205 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898664 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122272 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105397 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173588 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171647 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445561 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255871 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429311 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556008 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632955 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531601 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473552 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557782 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719016 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687568 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700172 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576311 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052502 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503438 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494793 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522024 s, 20.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28281 s, 19.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5452 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (52s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 13:46:14 (1777052774) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24205 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789525 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746409 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644413 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738768 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128754 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125307 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801572 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504808 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671762 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112902 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13776 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106781 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17624 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135291 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16843 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155816 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362514 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355933 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031202 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334368 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434618 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335836 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235221 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327137 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326085 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537375 s, 19.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534333 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378341 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409506 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384858 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043806 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532529 s, 19.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7759 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8966 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24311 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671801 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738532 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483592 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677655 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126733 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128487 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143501 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132631 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120129 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148387 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11656 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11405 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129904 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182792 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164938 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167245 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445625 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442374 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451545 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377764 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035428 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468595 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044752 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439644 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456699 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604451 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609532 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434873 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539079 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449526 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503337 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540557 s, 19.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7773 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9357 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (142s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 13:48:36 (1777052916) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24239 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228022 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507321 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912023 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124675 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124582 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103554 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827135 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104591 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795642 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137868 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16809 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126902 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124115 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114131 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140772 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176491 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505982 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597454 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422287 s, 24.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505295 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481677 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134499 s, 78.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762812 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753344 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600855 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683199 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441666 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043929 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470199 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106503 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544648 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030152 s, 348 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7984 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.942 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23233 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336619 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627025 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322656 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770461 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417019 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369885 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108551 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112182 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462602 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639821 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102535 s, 10.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105701 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110285 s, 9.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147187 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134004 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174771 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353189 s, 29.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317115 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426957 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111211 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399771 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483398 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319507 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417116 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380181 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051156 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915342 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495825 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388658 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399405 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388066 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469605 s, 22.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7604 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8515 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (140s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 13:50:56 (1777053056) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270439 s, 62.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518094 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504013 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585952 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455519 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667664 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566654 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557595 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302792 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507137 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803316 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918357 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853324 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713374 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781917 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850423 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137706 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381956 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040159 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347459 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412231 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416654 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544229 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488438 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02954 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562231 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457242 s, 22.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385153 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469331 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563731 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157627 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165341 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988378 s, 106 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21735 s, 20.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79813 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.59@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.59@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271623 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340569 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334884 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538431 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964354 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101834 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226895 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886403 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968232 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957118 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410429 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378655 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740815 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223335 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169939 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282621 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244739 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381278 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518438 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471385 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403763 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357145 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313932 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046385 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318442 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405686 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518334 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344529 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556204 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582638 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489639 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416862 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494915 s, 21.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32083 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80032 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.59@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257404 s, 65.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288407 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418041 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653387 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530091 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528658 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613588 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070772 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11335 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115388 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108187 s, 9.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108312 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095914 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704312 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082436 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924122 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957709 s, 10.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352942 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267224 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278082 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281628 s, 37.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456193 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397524 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311469 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332681 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335757 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367274 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370553 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429844 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401323 s, 26.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546691 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491995 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473971 s, 22.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27918 s, 19.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92754 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.457819 s, 36.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372498 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395196 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222818 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531353 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872253 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790115 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972028 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548936 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688571 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847284 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657607 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076037 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674768 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755224 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133598 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174401 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407706 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545126 s, 19.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461998 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447206 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480101 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490208 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701702 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568467 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639671 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620759 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485948 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693328 s, 15.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652523 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660478 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552428 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658256 s, 15.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88006 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77374 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06549 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79221 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.861584 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063317 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631057 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465367 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642678 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607533 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598553 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698662 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800946 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125096 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122374 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127916 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15481 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205531 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166404 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206533 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198964 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037417 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342357 s, 30.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392588 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577759 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402767 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540078 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649481 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044208 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493464 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379981 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581554 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577762 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051374 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563294 s, 18.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603166 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635473 s, 16.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6008 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5173 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.263344 s, 63.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388777 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585355 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752117 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726361 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082048 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979892 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956201 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691125 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991892 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795111 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909199 s, 11.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10279 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120105 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106862 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121568 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128415 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349596 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313245 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550538 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387159 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362498 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430128 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373506 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391594 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662883 s, 15.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555567 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464487 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532293 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501682 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576794 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485826 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548687 s, 19.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12407 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77794 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.446269 s, 37.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500205 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485782 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692572 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694258 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718717 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098591 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819051 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102973 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119181 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853652 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819702 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773095 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938256 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13799 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243118 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260894 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378267 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316144 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543636 s, 19.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034136 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388772 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100127 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461112 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185567 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427072 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564814 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577642 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658193 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451355 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495644 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808716 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556254 s, 18.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88805 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81658 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91499 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8125 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (369s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 13:57:05 (1777053425) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.282821 s, 59.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246174 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445888 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817312 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467121 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10042 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873596 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803183 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405322 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420137 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104678 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431966 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114896 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119928 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114866 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135531 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135084 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346778 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041139 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439287 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476955 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426122 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449021 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683523 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561858 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792849 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593738 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301523 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380278 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792591 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691588 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394676 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332584 s, 31.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98339 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77403 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (41s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 13:57:46 (1777053466) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.59@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.59@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273093 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316677 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386227 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542687 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046272 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402259 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517435 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828702 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309842 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725781 s, 14.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833586 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229814 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193028 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276909 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248369 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306856 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323138 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521663 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044314 s, 23.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450448 s, 23.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368403 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494639 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248668 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591394 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663411 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091038 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885782 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814705 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941376 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918604 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397324 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807184 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182852 s, 57.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.7165 s, 18.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79507 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (44s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 13:58:30 (1777053510) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.59@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.296925 s, 56.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628814 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730095 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618071 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818669 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138617 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14481 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151445 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110696 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874662 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142548 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112215 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129038 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094525 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129088 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136309 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15552 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537057 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633071 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580636 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749658 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646724 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650193 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730634 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754804 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724594 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831266 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085731 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876336 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109285 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916812 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904294 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889823 s, 11.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.21402 s, 20.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97865 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (39s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 13:59:10 (1777053550) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522485 s, 32.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420389 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736797 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640217 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767923 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690194 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105083 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961967 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110265 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135537 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10368 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127308 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117929 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14919 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111533 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188274 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18209 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047664 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426323 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297207 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376542 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299408 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357514 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346203 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428603 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538701 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514065 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507863 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521004 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501059 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05724 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057943 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581132 s, 18.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95721 s, 10.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81599 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.07555 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79396 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (83s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 14:00:33 (1777053633) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.932135 s, 18.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332083 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503537 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501268 s, 20.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469459 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537535 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377014 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073979 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521737 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124636 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124828 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141447 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146445 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170747 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154818 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201105 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1985 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040058 s, 26.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052868 s, 19.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279919 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364035 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362906 s, 28.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525594 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390619 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374784 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402871 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048769 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395253 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466095 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473055 s, 22.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424652 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749235 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497795 s, 21.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6339 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8683 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (69s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 14:01:42 (1777053702) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.298957 s, 56.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307561 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537837 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671503 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597683 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575631 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523315 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590048 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381161 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447201 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09699 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110929 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110172 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962623 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112029 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135256 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11607 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280412 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289034 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343165 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031421 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028949 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529295 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281905 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455458 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333055 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376083 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407965 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448773 s, 23.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679493 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598069 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437526 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503325 s, 20.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05236 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77224 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg159-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464056 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497426 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945623 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093118 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101734 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112045 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105358 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913515 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933047 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737932 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696483 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129044 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847318 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981303 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953073 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115694 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102371 s, 10.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573944 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849569 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573657 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682106 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718113 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646136 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739447 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623821 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770387 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708357 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561065 s, 18.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804269 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749741 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961617 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686073 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822623 s, 12.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88375 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81475 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87394 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79704 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (126s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 14:03:48 (1777053828) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping client oleg159-client.virtnet /mnt/lustre opts: Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg159-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg159-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.59,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg159-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.59,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg159-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.59,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63040/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.59,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg159-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Starting client oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Started clients oleg159-client.virtnet: 192.168.201.159@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Starting client oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Started clients oleg159-client.virtnet: 192.168.201.159@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee836cc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff97eebb6f7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee836cc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97eebb6f7000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.413175 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108526 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503454 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116763 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138839 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11447 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174709 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120224 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158703 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171709 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167399 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234946 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237638 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2026 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21927 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213864 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237223 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044666 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300937 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339752 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389379 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500128 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415842 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483033 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514357 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053805 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357999 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398099 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435026 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113254 s, 92.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399287 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152744 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311481 s, 337 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.94837 s, 17.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97697 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486765 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989871 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988692 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14006 s, 7.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623283 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138018 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120284 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135363 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146995 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126705 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208582 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25633 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239323 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250203 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278767 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289101 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234378 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97eebb6f7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129504 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133676 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118541 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124202 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124549 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127458 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162637 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142959 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137185 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139143 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144617 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144984 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148102 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148561 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150784 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153576 s, 6.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff97eebb6f7000.lock_unused_count=3 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg159-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg159-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.26828 s, 16.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84911 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (160s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 14:06:28 (1777053988) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg159-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (14s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 14:06:42 (1777054002) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (7s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 14:06:49 (1777054009) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg159-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.61125 s, 290 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg159-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0135 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (55s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 14:07:44 (1777054064) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.308269 s, 54.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05539 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10808 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110188 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13127 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878809 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578841 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105261 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111554 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129235 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97eebb6f7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184509 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160706 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231773 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181334 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286581 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227015 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318347 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496329 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644573 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545229 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854032 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677405 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760401 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099077 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800772 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582923 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103621 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689383 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714347 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795712 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835071 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805537 s, 13.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff97eebb6f7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff97eebb6f7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731961 s, 14.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14314 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93879 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.408691 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711713 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904653 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176527 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21621 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209073 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228165 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218286 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263215 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212281 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276796 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240629 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212619 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254121 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260623 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238611 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253618 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347405 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345816 s, 30.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408526 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585245 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511913 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468433 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499084 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042054 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526624 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454035 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461246 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04865 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047337 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049724 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050291 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507791 s, 20.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg159-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg159-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1141 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94798 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (72s) == sanityn test 77o: Changing rank should not panic ====== 14:08:56 (1777054136) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (8s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 14:09:04 (1777054144) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (75s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 14:10:19 (1777054219) mds.MDS.mdt.nrs_policies=tbf oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg159-client: oleg159-server: ssh exited with exit code 36 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg159-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (24s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 14:10:43 (1777054243) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 268725 kill md_thread xargs PASS 77r (50s) == sanityn test 77s: Check TBF LRU shrinker ============== 14:11:34 (1777054294) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (25s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 14:11:59 (1777054319) oleg159-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg159-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg159-client: oleg159-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (6s) == sanityn test 79: xattr: intent error ================== 14:12:05 (1777054325) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935986 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 14:12:06.000000000 -0400 Modify: 2026-04-24 14:12:06.000000000 -0400 Change: 2026-04-24 14:12:06.000000000 -0400 Birth: 2026-04-24 14:12:07.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (14s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 14:12:19 (1777054339) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 14:12:21 (1777054341) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 14:12:23 (1777054343) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 14:12:26 (1777054346) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 14:12:28 (1777054348) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 14:12:30 (1777054350) - open/close 1648 (time 1777054363.71 total 10.01 last 164.72) total: 2500 open/close in 15.55 seconds: 160.80 ops/second snapshot_time 1777054411.912150080 secs.nsecs start_time 1777054353.328347174 secs.nsecs elapsed_time 58.583802906 secs.nsecs open 2505 samples [usecs] 123 13528 2420230 2937836756 close 2504 samples [usecs] 80 2259 659552 246693598 mknod 2501 samples [usecs] 344 13390 2169311 2403972983 unlink 2500 samples [usecs] 457 33919 3322681 7721257963 mkdir 2 samples [usecs] 1672 3708 5380 16544848 rmdir 2 samples [usecs] 4480 6737 11217 65457569 rename 2500 samples [usecs] 453 33575 7938866 40217810368 getattr 3761 samples [usecs] 9 1713 181465 28154081 parallel_rename_file 2500 samples [usecs] 453 33575 7938866 40217810368 crossdir_rename 2500 samples [usecs] 453 33575 7938866 40217810368 PASS 81d (64s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 14:13:34 (1777054414) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7619 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7619 PASS 82 (4s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 14:13:38 (1777054418) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 14:13:40 (1777054420) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 85: Lustre API root cache race =========== 14:13:50 (1777054430) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1777054431 Finishing test test1 at 1777054432 PASS 85 (5s) == sanityn test 90: open/create and unlink striped directory ========================================================== 14:13:56 (1777054436) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 14:13:58 (1777054438) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 14:14:00 (1777054440) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 14:14:02 (1777054442) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6156 0x180c 0x240000400 1 6121 0x17e9 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6122 0x17ea 0x280000400 0 6157 0x180d 0x240000400 rc1=0 and rc2=0 PASS 93 (12s) == sanityn test 94: signal vs CP callback race =========== 14:14:14 (1777054454) osc.lustre-OST0000-osc-ffff97ee836cc000.idle_timeout=0 osc.lustre-OST0000-osc-ffff97eebb6f7000.idle_timeout=0 osc.lustre-OST0001-osc-ffff97ee836cc000.idle_timeout=0 osc.lustre-OST0001-osc-ffff97eebb6f7000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.103538 s, 79.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99775 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 287857 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0245241 s, 167 kB/s osc.lustre-OST0000-osc-ffff97ee836cc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff97eebb6f7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee836cc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97eebb6f7000.idle_timeout=debug PASS 94 (13s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 14:14:27 (1777054467) llite.lustre-ffff97ee836cc000.fast_read=0 llite.lustre-ffff97eebb6f7000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0301406 s, 69.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff97ee836cc000.fast_read=1 llite.lustre-ffff97eebb6f7000.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 14:14:44 (1777054484) llite.lustre-ffff97ee836cc000.fast_read=0 llite.lustre-ffff97eebb6f7000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00402776 s, 3.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff97ee836cc000.fast_read=1 llite.lustre-ffff97eebb6f7000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 14:14:54 (1777054494) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 14:14:56 (1777054496) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00469179 s, 873 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (5s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 14:15:00 (1777054500) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109347 s, 19.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (5s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 14:15:05 (1777054505) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117453 s, 17.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (5s) == sanityn test 100e: DoM: read on open and file size ==== 14:15:10 (1777054510) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00391987 s, 3.1 MB/s PASS 100e (4s) == sanityn test 101a: Discard DoM data on unlink ========= 14:15:14 (1777054514) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00449206 s, 912 kB/s PASS 101a (5s) == sanityn test 101b: Discard DoM data on rename ========= 14:15:19 (1777054519) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00398424 s, 1.0 MB/s PASS 101b (4s) == sanityn test 101c: Discard DoM data on close-unlink === 14:15:23 (1777054523) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00610992 s, 670 kB/s PASS 101c (5s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 14:15:30 (1777054530) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1777054531 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1777054534 Finishing test test23 at 1777054534 PASS 103 (10s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 14:15:40 (1777054540) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 14:15:42 (1777054542) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 14:16:07 (1777054567) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 14:16:09 (1777054569) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00952296 s, 108 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938521 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 14:16:09.000000000 -0400 Modify: 2026-04-24 14:16:09.000000000 -0400 Change: 2026-04-24 14:16:09.000000000 -0400 Birth: 2026-04-24 14:16:11.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938521 -rw-r--r-- 1 0 0 1777054571 1777054569 1777054569 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938521 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 14:16:09.000000000 -0400 Modify: 2026-04-24 14:16:09.000000000 -0400 Change: 2026-04-24 14:16:09.000000000 -0400 Birth: 2026-04-24 14:16:11.000000000 -0400 1777054569 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 14:16:13 (1777054573) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 14:16:16 (1777054576) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.175891 s, 59.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6165 0x1815 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 First grouplock blocks second one PASS 107a (6s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 14:16:22 (1777054582) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.174776 s, 60.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6129 0x17f1 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7619 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 14:16:32 (1777054592) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0066138 s, 1.5 MB/s llite.lustre-ffff97ee836cc000.hybrid_io=0 llite.lustre-ffff97eebb6f7000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.31887 s, 1.9 MB/s PASS 108a (8s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 14:16:40 (1777054600) 192.168.201.159@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg159-client.virtnet /mnt/lustre (opts:) 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 192.168.201.159@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg159-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 192.168.201.159@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg159-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 PASS 109 (41s) == sanityn test 110: do not grant another lock on resend ========================================================== 14:17:21 (1777054641) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-24 14:17:22.000000000 -0400 Modify: 2026-04-24 14:17:22.000000000 -0400 Change: 2026-04-24 14:17:22.000000000 -0400 Birth: 2026-04-24 14:17:23.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg159-client.virtnet oleg159-client.virtnet PASS 110 (167s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 14:20:08 (1777054808) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 14:20:10 (1777054810) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 14:20:12 (1777054812) lustre-OST0000-osc-ffff97ee886ea000 active. lustre-OST0001-osc-ffff97ee886ea000 active. lustre-OST0000-osc-ffff97ee886ea000 active. lustre-OST0001-osc-ffff97ee886ea000 active. MGC192.168.201.159@tcp active. PASS 113 (5s) == sanityn test 114: implicit default LMV inherit ======== 14:20:17 (1777054817) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 14:20:20 (1777054820) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 14:20:22 (1777054822) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 14:20:24 (1777054824) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0146913 s, 1.4 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0369667 s, 554 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (7s) == sanityn test 200: service remains healthy while able to process request ========================================================== 14:20:31 (1777054831) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1075 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1285 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1162 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1429 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1522 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1247 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1898 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1835 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1892 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1946 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2032 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2015 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.281 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.325 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3173 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3086 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97ee82c1f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ee82c1f800.ost_server_uuid in IDLE state after 0 sec PASS 200 (73s) cleanup: ====================================================== == sanityn test complete, duration 10724 sec ============= 14:21:45 (1777054905) === sanityn: start cleanup 14:21:46 (1777054906) === Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg159-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 14:23:37 (1777055017) ===