-----============= acceptance-small: sanityn ============----- Mon Mar 2 11:43:50 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg260-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 11:44:17 (1772469857) === Starting client oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Mount client oleg260-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg260-client.virtnet: executing check_config_client /mnt/lustre oleg260-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg260-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e8008cf6800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8e80202cc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e8008cf6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e80202cc800.idle_timeout=debug disable quota as required oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 11:45:01 (1772469901) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424293 s, 24.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7583] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 11:45:04 (1772469904) PASS 0 (12s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 11:45:16 (1772469916) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 11:45:25 (1772469925) -rw-r--r-- 1 root root 0 Mar 2 11:45 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 11:45:34 (1772469934) -rw-r--r-- 1 root root 0 Mar 2 11:45 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (9s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 11:45:42 (1772469942) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 11:45:50 (1772469950) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 11:45:58 (1772469958) total 1 -rwxrwxrwx 1 root root 0 Mar 2 11:45 f2a -rwxrwxrwx 1 root root 0 Mar 2 11:45 f2b total 1 -rwxrwxrwx 1 root root 0 Mar 2 11:45 f2a -rwxrwxrwx 1 root root 0 Mar 2 11:45 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.017672 s, 29.0 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 11:46:08 (1772469968) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 11:46:12 (1772469972) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.457297 s, 4.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (10s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 11:46:22 (1772469982) PASS 3 (10s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 11:46:32 (1772469992) PASS 4 (10s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 11:46:42 (1772470002) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 11:46:51 (1772470011) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (10s) == sanityn test 7: remove of open directory on other node ======================================================================= 11:47:01 (1772470021) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 11:47:10 (1772470030) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 11:47:18 (1772470038) PASS 9a (10s) == sanityn test 9b: append to striped sparse file ======== 11:47:28 (1772470048) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103018 s, 10.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.113852 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (10s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 11:47:38 (1772470058) 1+0 records in 1+0 records out 1 byte copied, 0.0343605 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0242677 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0192252 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0158709 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0096875 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010452 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0406161 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.014171 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.020119 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127397 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0180288 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.01439 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 11:47:49 (1772470069) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0229593 s, 134 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0595368 s, 68.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00323693 s, 949 kB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 11:47:57 (1772470077) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 11:48:06 (1772470086) warning: '-runtime' deprecated, use '-t runtime' instead - link 286 (time 1772470099.65 total 10.03 last 28.53) - link 552 (time 1772470109.67 total 20.05 last 26.54) - link 863 (time 1772470119.70 total 30.08 last 31.00) - link 1096 (time 1772470129.75 total 40.13 last 23.18) - link 1342 (time 1772470139.79 total 50.17 last 24.52) - link 1630 (time 1772470149.81 total 60.19 last 28.74) - link 1859 (time 1772470159.81 total 70.19 last 22.89) - link 2114 (time 1772470169.83 total 80.21 last 25.45) - link 2393 (time 1772470179.84 total 90.22 last 27.88) total: 2621 link in 100.04 seconds: 26.20 ops/second using seed 1335472845 running for 100 seconds - stat 3667 (time 1772470101 ; total 11 ; last 11) - stat 8494 (time 1772470112 ; total 22 ; last 11) - stat 10000 (time 1772470115 ; total 25 ; last 3) - stat 14723 (time 1772470126 ; total 36 ; last 11) - stat 18753 (time 1772470137 ; total 47 ; last 11) - stat 20000 (time 1772470140 ; total 50 ; last 3) - stat 22457 (time 1772470151 ; total 61 ; last 11) - stat 24493 (time 1772470162 ; total 72 ; last 11) - stat 26699 (time 1772470173 ; total 83 ; last 11) - stat 28810 (time 1772470184 ; total 94 ; last 11) total: 29973 stats in 100 seconds: 299.730011 stats/second - unlinked 0 (time 1772470140 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2621) error: No such file or directory total: 2620 unlinks in 82 seconds: 31.951220 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21401) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21403) - No such process PASS 12 (143s) == sanityn test 13: test directory page revocation ======= 11:50:29 (1772470229) f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 11:50:40 (1772470240) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22588 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 11:50:48 (1772470248) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23168 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 11:50:57 (1772470257) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23747 Terminated $DIR1/$tdir/sleep 60 PASS 14b (10s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 11:51:08 (1772470268) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24330 Terminated $DIR1/$tdir/sleep 60 PASS 14c (10s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 11:51:18 (1772470278) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24912 Terminated $DIR1/$tdir/sleep 60 PASS 14d (12s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 11:51:30 (1772470290) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (53s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 11:52:24 (1772470344) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.678459 s, 12.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.438419 s, 19.1 MB/s Chance of close/open is 1 in 50 Seed set to 348 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1772470350.049133 write 0x3ec1f0 thru 0x3f0438 (0x4249 bytes) truncating to largest ever: 0x328853 000003[0] 1772470350.109816 trunc from 0x3f0439 to 0x328852 (0xc7be6 bytes) 000005[1] 1772470350.202988 write 0x747b1c thru 0x74ed5d (0x7242 bytes) 000006[1] 1772470350.268351 mapwrite 0x053299 thru 0x0558df (0x2647 bytes) 000009[0] 1772470350.888475 write 0x1019ad thru 0x10bc6a (0xa2be bytes) 000011[0] 1772470351.015805 mapread 0x1f7965 thru 0x205d5f (0xe3fb bytes) 000012[0] 1772470351.104763 trunc from 0x74ed5e to 0x04ec7e (0x7000df bytes) 000014[1] 1772470351.251971 write 0x457c41 thru 0x45bd8a (0x414a bytes) 000016[0] 1772470351.327074 read 0x4017ca thru 0x407837 (0x606e bytes) 000017[1] 1772470351.405024 write 0x86b741 thru 0x86d42d (0x1ced bytes) 000018[0] 1772470351.463385 read 0x4dbae0 thru 0x4e8969 (0xce8a bytes) 000019[0] 1772470351.692947 mapwrite 0x31a387 thru 0x3278c9 (0xd543 bytes) 000020[0] 1772470352.136164 read 0x20f9c3 thru 0x21c46c (0xcaaa bytes) 000021[0] 1772470352.160303 read 0x29fc2b thru 0x2ab761 (0xbb37 bytes) 000023[1] 1772470352.280110 write 0x99e5b1 thru 0x99f6b1 (0x1101 bytes) 000024[1] 1772470352.328141 mapwrite 0x048892 thru 0x05409a (0xb809 bytes) 000027[0] 1772470352.505149 read 0x8d8e2a thru 0x8dde85 (0x505c bytes) 000029[0] 1772470352.550854 write 0x219f49 thru 0x21c982 (0x2a3a bytes) truncating to largest ever: 0x88f35e 000036[1] 1772470352.662728 trunc from 0x99f6b2 to 0x88f35d (0x110354 bytes) 000038[0] 1772470352.755169 mapread 0x5fc2d7 thru 0x60b0a0 (0xedca bytes) 000039[1] 1772470352.877199 read 0x1c1645 thru 0x1c7184 (0x5b40 bytes) 000040[1] 1772470352.909039 trunc from 0x88f35e to 0x5aa950 (0x2e4a0d bytes) truncating to largest ever: 0x9d1d3b 000041[0] 1772470352.992395 trunc from 0x5aa951 to 0x9d1d3a (0x4273ea bytes) 000043[1] 1772470353.085017 read 0x1902e8 thru 0x19ae4d (0xab66 bytes) 000046[1] 1772470353.135092 mapread 0x525877 thru 0x52e6de (0x8e68 bytes) 000050[0] 1772470353.191477 trunc from 0x9d1d3b to 0x361cf1 (0x670049 bytes) 000051[1] 1772470353.256757 mapwrite 0x4ed4c6 thru 0x4f1fa8 (0x4ae3 bytes) 000053[1] 1772470353.431118 mapwrite 0x28eafd thru 0x29b33e (0xc842 bytes) 000054[0] 1772470353.573387 read 0x474ed3 thru 0x47577a (0x8a8 bytes) 000055[0] 1772470353.618658 trunc from 0x4f1fa9 to 0x30790c (0x1ea69c bytes) 000056[0] 1772470353.685833 mapwrite 0x2b255d thru 0x2ba7dc (0x8280 bytes) 000061[1] 1772470353.996484 trunc from 0x30790d to 0x0468f5 (0x2c1017 bytes) 000062[0] 1772470354.102943 read 0x02b0ad thru 0x031f36 (0x6e8a bytes) 000063[0] 1772470354.163832 write 0x2dd133 thru 0x2e6ed9 (0x9da7 bytes) 000064[1] 1772470354.209841 mapread 0x15caea thru 0x1697fd (0xcd14 bytes) 000065[1] 1772470354.345362 mapwrite 0x95dc38 thru 0x96b705 (0xdace bytes) 000066[1] 1772470354.575862 mapread 0x284f5f thru 0x288a24 (0x3ac6 bytes) 000071[1] 1772470354.694082 trunc from 0x96b706 to 0x778ede (0x1f2827 bytes) 000073[1] 1772470354.785769 trunc from 0x778edf to 0x83e54e (0xc5670 bytes) 000075[1] 1772470354.822197 mapread 0x15d1b9 thru 0x16cb6b (0xf9b3 bytes) 000078[1] 1772470354.934988 mapwrite 0x58d065 thru 0x595a13 (0x89af bytes) 000083[1] 1772470355.154596 write 0x8974e0 thru 0x89dd7a (0x689b bytes) 000085[0] 1772470355.191946 read 0x7f0c72 thru 0x7f96e8 (0x8a77 bytes) 000089[1] 1772470355.277036 write 0x57765a thru 0x578312 (0xcb9 bytes) 000091[0] 1772470355.342435 read 0x83c400 thru 0x846027 (0x9c28 bytes) 000093[1] 1772470355.417907 trunc from 0x89dd7b to 0x173b5d (0x72a21d bytes) 000095[0] 1772470355.509469 mapread 0x040ee3 thru 0x049e9e (0x8fbc bytes) 000096[0] 1772470355.577869 read 0x105b2d thru 0x10c266 (0x673a bytes) 000102[1] 1772470355.673698 mapwrite 0x2a8725 thru 0x2b7706 (0xefe2 bytes) 000103[0] 1772470355.814936 mapwrite 0x53516d thru 0x5358ff (0x793 bytes) 000104[1] 1772470356.047075 mapwrite 0x504b4f thru 0x510462 (0xb914 bytes) 000107[1] 1772470356.281791 write 0x60e6ca thru 0x61e2d6 (0xfc0d bytes) 000110[0] 1772470356.408574 mapread 0x0d9f5b thru 0x0e1352 (0x73f8 bytes) 000111[0] 1772470356.557199 mapwrite 0x389643 thru 0x38bad0 (0x248e bytes) 000114[0] 1772470356.845666 mapread 0x0e62a4 thru 0x0f29ba (0xc717 bytes) 000116[0] 1772470357.045163 mapwrite 0x8d0ecc thru 0x8dd68d (0xc7c2 bytes) 000119[0] 1772470357.364312 read 0x8560c4 thru 0x85efac (0x8ee9 bytes) 000120[1] 1772470357.365287 mapwrite 0x8725b3 thru 0x878933 (0x6381 bytes) 000121[1] 1772470357.632619 mapread 0x62b863 thru 0x637e7e (0xc61c bytes) 000122[0] 1772470357.808940 mapread 0x19a4ec thru 0x19b660 (0x1175 bytes) 000123[1] 1772470357.866038 mapread 0x3fb461 thru 0x408c2f (0xd7cf bytes) 000125[1] 1772470358.009036 write 0x429068 thru 0x42e239 (0x51d2 bytes) 000129[0] 1772470358.244408 write 0x2e592c thru 0x2e7679 (0x1d4e bytes) 000130[1] 1772470358.324396 trunc from 0x8dd68e to 0x8398b3 (0xa3dda bytes) 000131[1] 1772470358.376975 mapwrite 0x259276 thru 0x25b4a1 (0x222c bytes) 000133[1] 1772470358.554876 mapwrite 0x9063f1 thru 0x911715 (0xb325 bytes) 000134[0] 1772470358.766145 write 0x371a36 thru 0x3721da (0x7a5 bytes) 000135[0] 1772470358.825375 mapwrite 0x0d2f38 thru 0x0e0715 (0xd7de bytes) 000138[0] 1772470359.163803 write 0x34f89c thru 0x3559cf (0x6134 bytes) 000141[0] 1772470359.312156 trunc from 0x911716 to 0x0338a5 (0x8dde70 bytes) 000142[1] 1772470359.437092 mapread 0x026983 thru 0x02ed7c (0x83fa bytes) 000143[1] 1772470359.682928 write 0x34851a thru 0x354eab (0xc992 bytes) 000145[0] 1772470359.779798 mapread 0x317037 thru 0x31b579 (0x4543 bytes) 000146[1] 1772470359.913089 write 0x4ef4ef thru 0x4f563c (0x614e bytes) 000148[1] 1772470359.962236 mapread 0x3952e1 thru 0x3a2a3c (0xd75c bytes) 000150[0] 1772470360.143729 trunc from 0x4f563d to 0x4e6e25 (0xe817 bytes) 000151[0] 1772470360.310793 trunc from 0x4e6e26 to 0x4634f5 (0x83930 bytes) 000157[1] 1772470360.520266 trunc from 0x4634f6 to 0x097eed (0x3cb608 bytes) 000163[1] 1772470360.682680 mapread 0x05a6f2 thru 0x061a0e (0x731d bytes) 000169[1] 1772470360.842365 mapwrite 0x3dc73c thru 0x3e6d2f (0xa5f4 bytes) 000170[0] 1772470361.071217 write 0x4a3b7f thru 0x4b1939 (0xddbb bytes) 000179[1] 1772470361.313801 write 0x3e29c8 thru 0x3ed329 (0xa962 bytes) 000182[0] 1772470361.364999 write 0x0959f9 thru 0x0a4ec3 (0xf4cb bytes) 000183[1] 1772470361.406104 mapread 0x2f6ca2 thru 0x2f87f2 (0x1b51 bytes) 000185[0] 1772470361.568628 trunc from 0x4b193a to 0x65598e (0x1a4055 bytes) 000186[1] 1772470361.626743 mapwrite 0x5b719e thru 0x5c1e34 (0xac97 bytes) 000187[0] 1772470361.938059 mapread 0x44318f thru 0x4441d3 (0x1045 bytes) 000189[0] 1772470362.056986 read 0x3d8aff thru 0x3e19c8 (0x8eca bytes) 000192[0] 1772470362.146301 read 0x6049a3 thru 0x612861 (0xdebf bytes) 000193[1] 1772470362.185860 read 0x62e2bc thru 0x6331c9 (0x4f0e bytes) 000194[0] 1772470362.229747 trunc from 0x65598f to 0x26976c (0x3ec222 bytes) 000195[1] 1772470362.296364 write 0x41b494 thru 0x426bf0 (0xb75d bytes) 000196[1] 1772470362.363900 mapread 0x1adc8e thru 0x1bdc6b (0xffde bytes) 000203[1] 1772470362.538735 write 0x76196c thru 0x76848b (0x6b20 bytes) 000206[1] 1772470362.580051 read 0x0f53bd thru 0x0f8cee (0x3932 bytes) 000210[0] 1772470362.772644 mapread 0x4fa8ef thru 0x509cab (0xf3bd bytes) 000211[1] 1772470362.893600 read 0x640eb1 thru 0x64997b (0x8acb bytes) 000212[1] 1772470362.914057 read 0x153745 thru 0x154b29 (0x13e5 bytes) 000218[1] 1772470362.925021 mapwrite 0x5a900a thru 0x5afa79 (0x6a70 bytes) 000221[0] 1772470363.105675 read 0x4fcb28 thru 0x505a62 (0x8f3b bytes) 000223[1] 1772470363.156022 trunc from 0x76848c to 0x1e5bd9 (0x5828b2 bytes) 000224[0] 1772470363.276463 read 0x0b197e thru 0x0bf9d7 (0xe05a bytes) 000225[1] 1772470363.361992 write 0x9aa97c thru 0x9b1047 (0x66cc bytes) 000226[0] 1772470363.414966 trunc from 0x9b1048 to 0x6ccb7a (0x2e44cd bytes) 000227[0] 1772470363.522510 mapwrite 0x55fd3b thru 0x560315 (0x5db bytes) 000229[0] 1772470363.773143 mapwrite 0x35986f thru 0x368375 (0xeb07 bytes) 000230[1] 1772470363.884500 write 0x33fb1b thru 0x34a24e (0xa734 bytes) 000231[1] 1772470363.961672 read 0x38acda thru 0x393759 (0x8a80 bytes) 000233[0] 1772470364.016910 mapwrite 0x18b384 thru 0x18ec12 (0x388f bytes) 000234[1] 1772470364.170074 mapread 0x025306 thru 0x032ce5 (0xd9e0 bytes) 000236[1] 1772470364.269714 mapwrite 0x400c2a thru 0x410990 (0xfd67 bytes) 000237[1] 1772470364.393451 read 0x33a317 thru 0x3441b6 (0x9ea0 bytes) 000239[1] 1772470364.419586 write 0x8bda5d thru 0x8c4726 (0x6cca bytes) 000240[1] 1772470364.440618 mapwrite 0x4702e1 thru 0x4716b5 (0x13d5 bytes) 000243[0] 1772470364.523189 write 0x6ed403 thru 0x6f5f29 (0x8b27 bytes) 000244[0] 1772470364.553579 read 0x775dec thru 0x77b8d7 (0x5aec bytes) 000249[0] 1772470364.833406 mapwrite 0x838899 thru 0x83a42e (0x1b96 bytes) 000250[1] 1772470365.092611 read 0x2a97af thru 0x2ad500 (0x3d52 bytes) 000251[0] 1772470365.189038 write 0x02c8fd thru 0x02ec76 (0x237a bytes) 000252[0] 1772470365.240479 read 0x470534 thru 0x47e03e (0xdb0b bytes) 000253[0] 1772470365.304018 mapwrite 0x08e8a0 thru 0x099a9c (0xb1fd bytes) 000255[0] 1772470365.398939 read 0x228caf thru 0x2299f7 (0xd49 bytes) 000256[1] 1772470365.439459 mapread 0x7ae676 thru 0x7b1c5a (0x35e5 bytes) 000259[1] 1772470365.584776 trunc from 0x8c4727 to 0x5cac51 (0x2f9ad5 bytes) 000261[1] 1772470365.728292 mapwrite 0x864b2a thru 0x870513 (0xb9ea bytes) 000263[1] 1772470365.999022 trunc from 0x870514 to 0x00f1d6 (0x86133d bytes) 000266[0] 1772470366.117190 mapwrite 0x295f29 thru 0x29b325 (0x53fd bytes) 000267[1] 1772470366.361079 mapwrite 0x1f16ae thru 0x1f5344 (0x3c97 bytes) 000269[0] 1772470366.600644 trunc from 0x29b326 to 0x35b2bd (0xbff98 bytes) 000270[1] 1772470366.697705 trunc from 0x35b2be to 0x175fbc (0x1e5301 bytes) 000271[1] 1772470366.754720 write 0x900879 thru 0x90a6a0 (0x9e28 bytes) 000274[1] 1772470366.797673 read 0x8ad025 thru 0x8b83a7 (0xb383 bytes) 000275[0] 1772470366.846967 mapread 0x497e59 thru 0x4a48c6 (0xca6e bytes) 000277[0] 1772470367.109980 write 0x51683e thru 0x52249e (0xbc61 bytes) 000278[1] 1772470367.143924 read 0x765e91 thru 0x77325d (0xd3cd bytes) 000281[0] 1772470367.306960 trunc from 0x90a6a1 to 0x0641cb (0x8a64d5 bytes) 000282[0] 1772470367.419815 write 0x1f3692 thru 0x1ffe3d (0xc7ac bytes) 000286[0] 1772470367.497784 mapwrite 0x136e2f thru 0x13cc3e (0x5e10 bytes) 000288[1] 1772470367.759590 trunc from 0x1ffe3e to 0x0b5418 (0x14aa25 bytes) 000289[1] 1772470367.962843 read 0x088331 thru 0x08c1f2 (0x3ec2 bytes) 000290[1] 1772470368.021983 write 0x652fdb thru 0x6586d5 (0x56fb bytes) 000291[0] 1772470368.073746 mapread 0x0fc582 thru 0x0fd3a0 (0xe1f bytes) 000293[0] 1772470368.247988 read 0x60fed2 thru 0x61087b (0x9aa bytes) 000297[1] 1772470368.326060 trunc from 0x6586d6 to 0x1daaf8 (0x47dbdd bytes) 000299[1] 1772470368.497062 mapwrite 0x90ce2c thru 0x916036 (0x920b bytes) 000300[0] 1772470368.802602 write 0x6509d9 thru 0x65e7b6 (0xddde bytes) 000304[0] 1772470368.944987 mapread 0x551f19 thru 0x5571de (0x52c6 bytes) 000305[1] 1772470369.036660 mapwrite 0x70a24f thru 0x711d8b (0x7b3d bytes) 000306[0] 1772470369.223968 mapread 0x7338c7 thru 0x736dea (0x3524 bytes) 000307[1] 1772470369.308963 trunc from 0x916037 to 0x791625 (0x184a11 bytes) 000311[1] 1772470369.477502 mapwrite 0x056b37 thru 0x05704e (0x518 bytes) 000312[1] 1772470369.563906 read 0x28f73a thru 0x2936f2 (0x3fb9 bytes) 000313[0] 1772470369.570671 mapread 0x2cac8d thru 0x2cf7ce (0x4b42 bytes) 000315[1] 1772470369.644828 write 0x65ea9e thru 0x6691c5 (0xa728 bytes) 000318[1] 1772470369.661237 mapwrite 0x72b9af thru 0x73b73a (0xfd8c bytes) 000319[0] 1772470369.779249 mapread 0x02ba63 thru 0x039bd8 (0xe176 bytes) 000320[0] 1772470369.818954 mapwrite 0x4cd315 thru 0x4d4f3f (0x7c2b bytes) 000321[0] 1772470369.940725 mapwrite 0x909f8d thru 0x9144a5 (0xa519 bytes) 000322[1] 1772470370.074728 write 0x41c2ba thru 0x429ad8 (0xd81f bytes) 000324[0] 1772470370.143692 mapread 0x1a8e36 thru 0x1b68d8 (0xdaa3 bytes) 000326[0] 1772470370.223629 mapread 0x8304d1 thru 0x831168 (0xc98 bytes) 000327[0] 1772470370.259441 read 0x15dffd thru 0x164052 (0x6056 bytes) 000330[0] 1772470370.330654 trunc from 0x9144a6 to 0x23c832 (0x6d7c73 bytes) 000334[1] 1772470370.462158 mapread 0x10c164 thru 0x1196bc (0xd559 bytes) 000335[1] 1772470370.538588 mapread 0x0a82f5 thru 0x0b4446 (0xc152 bytes) 000336[0] 1772470370.581457 write 0x4480b9 thru 0x44f31a (0x7262 bytes) 000340[1] 1772470370.654795 mapread 0x190161 thru 0x193b90 (0x3a30 bytes) 000342[1] 1772470370.701967 trunc from 0x44f31b to 0x38fae2 (0xbf838 bytes) 000343[1] 1772470370.772250 mapread 0x303fd0 thru 0x30dd1f (0x9d50 bytes) 000344[1] 1772470370.851918 trunc from 0x38fae3 to 0x47b9e5 (0xebf03 bytes) 000346[1] 1772470370.885971 trunc from 0x47b9e6 to 0x97cace (0x5010e9 bytes) 000347[1] 1772470370.943099 trunc from 0x97cacf to 0x7d0723 (0x1ac3ab bytes) 000348[0] 1772470370.977092 mapwrite 0x481c7a thru 0x48c821 (0xaba8 bytes) 000350[0] 1772470371.158672 read 0x5db44d thru 0x5e90a3 (0xdc57 bytes) 000352[1] 1772470371.242725 write 0x246036 thru 0x24788e (0x1859 bytes) 000353[0] 1772470371.295755 mapread 0x574fd7 thru 0x5759fe (0xa28 bytes) 000358[0] 1772470371.430107 mapread 0x3074e3 thru 0x313d6f (0xc88d bytes) 000361[1] 1772470371.607468 mapread 0x74f843 thru 0x756ab4 (0x7272 bytes) 000362[0] 1772470371.754503 write 0x553362 thru 0x558141 (0x4de0 bytes) 000364[1] 1772470371.872725 mapread 0x212e4f thru 0x214524 (0x16d6 bytes) 000366[0] 1772470371.929487 read 0x2c7dbe thru 0x2cf2af (0x74f2 bytes) 000368[0] 1772470371.951963 trunc from 0x7d0724 to 0x0b0b03 (0x71fc20 bytes) 000370[1] 1772470372.060062 mapread 0x049796 thru 0x04fa6e (0x62d9 bytes) 000373[0] 1772470372.118640 mapwrite 0x4f98d0 thru 0x505814 (0xbf45 bytes) 000375[0] 1772470372.276219 mapwrite 0x5e8f77 thru 0x5eb268 (0x22f2 bytes) 000376[1] 1772470372.416134 trunc from 0x5eb269 to 0x16118d (0x48a0db bytes) 000378[0] 1772470372.528794 trunc from 0x16118e to 0x964bb6 (0x803a29 bytes) 000379[0] 1772470372.622802 write 0x3eb196 thru 0x3f5e2d (0xac98 bytes) 000384[1] 1772470372.761934 trunc from 0x964bb7 to 0x4347ad (0x530409 bytes) 000387[0] 1772470372.852204 trunc from 0x4347ae to 0x400b52 (0x33c5b bytes) 000388[0] 1772470372.936776 write 0x7013a0 thru 0x70a809 (0x946a bytes) 000389[1] 1772470372.991050 trunc from 0x70a80a to 0x520ac0 (0x1e9d49 bytes) 000393[1] 1772470373.188664 mapwrite 0x047339 thru 0x050af6 (0x97be bytes) 000394[0] 1772470373.383771 write 0x9e1fd5 thru 0x9e7b0e (0x5b3a bytes) 000395[1] 1772470373.463867 mapread 0x175236 thru 0x17f10c (0x9ed7 bytes) 000397[0] 1772470373.625934 trunc from 0x9e7b0f to 0x7403dd (0x2a7731 bytes) 000399[0] 1772470373.772680 mapwrite 0x571088 thru 0x574df5 (0x3d6e bytes) 000400[1] 1772470373.926641 read 0x2e67e9 thru 0x2f3722 (0xcf3a bytes) 000401[0] 1772470373.993728 read 0x131f79 thru 0x1413ff (0xf487 bytes) 000405[0] 1772470374.021504 trunc from 0x7403de to 0x266e62 (0x4d957b bytes) 000416[0] 1772470374.483432 mapwrite 0x0ce03c thru 0x0d8c51 (0xac16 bytes) 000419[1] 1772470374.817675 trunc from 0x266e63 to 0x879b0c (0x612caa bytes) 000420[1] 1772470375.064915 read 0x54050a thru 0x547ad8 (0x75cf bytes) 000421[1] 1772470375.123871 read 0x51914d thru 0x5265c4 (0xd478 bytes) 000423[1] 1772470375.226769 mapread 0x638de1 thru 0x63d38d (0x45ad bytes) 000424[1] 1772470375.290813 write 0x22f949 thru 0x232322 (0x29da bytes) 000425[1] 1772470375.346749 mapread 0x2bb4c2 thru 0x2bb8fc (0x43b bytes) 000428[1] 1772470375.399726 mapread 0x4aa3aa thru 0x4b697f (0xc5d6 bytes) 000429[0] 1772470375.473751 read 0x66c409 thru 0x676cb8 (0xa8b0 bytes) 000432[0] 1772470375.550971 trunc from 0x879b0d to 0x83bdd0 (0x3dd3c bytes) 000433[1] 1772470375.758096 mapread 0x237249 thru 0x243c24 (0xc9dc bytes) 000434[1] 1772470375.993898 mapread 0x5fa116 thru 0x5fa9ac (0x897 bytes) 000435[0] 1772470376.114231 write 0x771aa9 thru 0x779323 (0x787b bytes) 000436[1] 1772470376.174397 mapwrite 0x728c15 thru 0x7352e8 (0xc6d4 bytes) 000438[0] 1772470376.450678 mapread 0x5afb49 thru 0x5b8c9e (0x9156 bytes) 000439[1] 1772470376.542674 write 0x94333e thru 0x9507c5 (0xd488 bytes) 000441[0] 1772470376.631121 read 0x375267 thru 0x37ce69 (0x7c03 bytes) 000444[1] 1772470376.708421 trunc from 0x9507c6 to 0x6fa478 (0x25634d bytes) 000445[0] 1772470376.855654 read 0x3223e8 thru 0x331450 (0xf069 bytes) 000448[1] 1772470377.025198 mapread 0x11749f thru 0x11e88f (0x73f1 bytes) 000451[0] 1772470377.149642 trunc from 0x6fa479 to 0x55e637 (0x19be41 bytes) 000452[1] 1772470377.274215 write 0x697014 thru 0x6982d0 (0x12bd bytes) 000454[1] 1772470377.355591 trunc from 0x6982d1 to 0x4d6679 (0x1c1c57 bytes) 000455[1] 1772470377.423487 read 0x3f94c8 thru 0x404910 (0xb449 bytes) 000456[1] 1772470377.474093 mapread 0x3a58b4 thru 0x3a7dd2 (0x251f bytes) 000457[1] 1772470377.540022 mapread 0x3d1f5a thru 0x3dbfad (0xa054 bytes) 000458[0] 1772470377.591031 mapwrite 0x072013 thru 0x076329 (0x4317 bytes) 000463[1] 1772470377.887318 trunc from 0x4d667a to 0x097e96 (0x43e7e3 bytes) 000464[0] 1772470377.944819 write 0x0848b8 thru 0x089871 (0x4fba bytes) 000466[1] 1772470378.039165 trunc from 0x097e97 to 0x23fb2d (0x1a7c97 bytes) 000467[0] 1772470378.097692 write 0x10918a thru 0x11099c (0x7813 bytes) 000468[0] 1772470378.144412 mapwrite 0x2ed854 thru 0x2f2db4 (0x5561 bytes) 000470[1] 1772470378.403648 read 0x0d153c thru 0x0d20e2 (0xba7 bytes) 000472[1] 1772470378.647580 mapwrite 0x0f1f43 thru 0x0fa0ca (0x8188 bytes) 000477[0] 1772470379.208812 mapread 0x0fd46f thru 0x10044c (0x2fde bytes) 000479[1] 1772470379.369581 mapwrite 0x81b610 thru 0x821711 (0x6102 bytes) 000480[1] 1772470379.799925 mapwrite 0x8e6dc8 thru 0x8e9e6b (0x30a4 bytes) 000481[1] 1772470379.975477 read 0x2660db thru 0x26a87e (0x47a4 bytes) 000486[1] 1772470380.064247 write 0x0834bb thru 0x0887a3 (0x52e9 bytes) 000487[1] 1772470380.133496 mapwrite 0x99cdfd thru 0x9a1634 (0x4838 bytes) 000488[0] 1772470380.284814 read 0x1c71fe thru 0x1ccd8d (0x5b90 bytes) 000489[1] 1772470380.358371 trunc from 0x9a1635 to 0x3b3534 (0x5ee100 bytes) 000490[1] 1772470380.464569 read 0x14647d thru 0x1546a8 (0xe22c bytes) 000491[0] 1772470380.647361 mapread 0x008c57 thru 0x00ac98 (0x2042 bytes) 000492[1] 1772470380.785510 write 0x6dbfe2 thru 0x6e2ac5 (0x6ae4 bytes) 000493[1] 1772470380.815500 mapread 0x1e2600 thru 0x1f0943 (0xe344 bytes) 000494[0] 1772470380.954594 mapwrite 0x2db2a6 thru 0x2dd8cf (0x262a bytes) 000496[0] 1772470381.229142 write 0x4d7a6d thru 0x4e7399 (0xf92d bytes) 000497[0] 1772470381.332471 mapread 0x4cff07 thru 0x4d5e72 (0x5f6c bytes) 000498[1] 1772470381.452780 trunc from 0x6e2ac6 to 0x0ff997 (0x5e312e bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 381 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x10b0aa 000006[0] 1772470382.712832 trunc from 00000000 to 0x10b0a9 (0x10b0aa bytes) 000007[0] 1772470382.825731 mapwrite 0x3bc000 thru 0x3c6fc3 (0xafc4 bytes) 000009[0] 1772470383.042823 read 0x259000 thru 0x262afd (0x9afe bytes) truncating to largest ever: 0x53ff38 000010[1] 1772470383.109001 trunc from 0x3c6fc4 to 0x53ff37 (0x178f74 bytes) truncating to largest ever: 0x900756 000011[1] 1772470383.274865 trunc from 0x53ff38 to 0x900755 (0x3c081e bytes) 000012[0] 1772470383.317839 read 0x89a000 thru 0x8a0ae2 (0x6ae3 bytes) 000013[1] 1772470383.361048 read_OD 0x0dd000 thru 0x0defff (0x2000 bytes) 000014[0] 1772470383.380779 mapread 0x4ae000 thru 0x4aec00 (0xc01 bytes) 000015[0] 1772470383.391848 read 0x6e2000 thru 0x6e2404 (0x405 bytes) 000017[0] 1772470383.401807 mapwrite 0x84b000 thru 0x852a8d (0x7a8e bytes) 000018[1] 1772470383.540806 trunc from 0x900756 to 0x3b2665 (0x54e0f0 bytes) 000019[0] 1772470383.599651 mapread 0x35e000 thru 0x363fb6 (0x5fb7 bytes) 000021[0] 1772470383.693487 trunc from 0x3b2666 to 0x191af4 (0x220b71 bytes) 000023[1] 1772470383.778467 read_OD 0x183000 thru 0x190fff (0xe000 bytes) 000024[1] 1772470383.829787 write_OD 0x2ef000 thru 0x2f9fff (0xb000 bytes) 000030[0] 1772470383.923742 mapwrite 0x580000 thru 0x588f36 (0x8f37 bytes) 000032[1] 1772470384.149219 read_OD 0x077000 thru 0x085fff (0xf000 bytes) 000033[0] 1772470384.200239 write 0x03c000 thru 0x043370 (0x7371 bytes) 000035[1] 1772470384.276596 mapwrite 0x7db000 thru 0x7def34 (0x3f35 bytes) 000036[1] 1772470384.433130 mapread 0x7de000 thru 0x7deb79 (0xb7a bytes) 000037[1] 1772470384.472197 trunc from 0x7def35 to 0x79fd06 (0x3f22e bytes) 000039[1] 1772470384.555715 write_OD 0x85a000 thru 0x863fff (0xa000 bytes) 000040[1] 1772470384.646017 mapread 0x0a6000 thru 0x0aa426 (0x4427 bytes) 000043[0] 1772470384.716202 mapread 0x129000 thru 0x137391 (0xe392 bytes) 000044[1] 1772470384.775147 read_OD 0x574000 thru 0x581fff (0xe000 bytes) 000045[1] 1772470384.817842 trunc from 0x864000 to 0x2810ed (0x5e2f12 bytes) 000046[0] 1772470384.949222 mapwrite 0x438000 thru 0x438fbb (0xfbc bytes) 000047[0] 1772470385.115498 read 0x290000 thru 0x29cb6b (0xcb6c bytes) 000052[0] 1772470385.236695 mapwrite 0x0b4000 thru 0x0bfe4c (0xbe4d bytes) 000053[0] 1772470385.410739 read 0x302000 thru 0x30200d (0xe bytes) 000054[1] 1772470385.431615 read_OD 0x0d1000 thru 0x0defff (0xe000 bytes) 000055[0] 1772470385.521031 mapread 0x154000 thru 0x15cb58 (0x8b59 bytes) 000057[0] 1772470385.594153 trunc from 0x438fbc to 0x7705de (0x337623 bytes) 000058[0] 1772470385.743723 mapwrite 0x99c000 thru 0x9a9337 (0xd338 bytes) 000059[1] 1772470385.924376 mapread 0x152000 thru 0x153e1e (0x1e1f bytes) 000064[1] 1772470386.017747 mapread 0x378000 thru 0x3783c0 (0x3c1 bytes) 000066[0] 1772470386.029309 trunc from 0x9a9338 to 0x80c012 (0x19d325 bytes) 000067[0] 1772470386.101134 trunc from 0x80c013 to 0x164c42 (0x6a73d0 bytes) 000069[1] 1772470386.151280 trunc from 0x164c43 to 0x162737 (0x250b bytes) 000072[0] 1772470386.251355 write 0x4c4000 thru 0x4d0a07 (0xca08 bytes) 000073[0] 1772470386.290552 mapwrite 0x5eb000 thru 0x5f8a34 (0xda35 bytes) 000075[1] 1772470386.548139 write_OD 0x5a5000 thru 0x5a8fff (0x4000 bytes) 000079[0] 1772470386.692864 write 0x94a000 thru 0x954301 (0xa302 bytes) 000080[0] 1772470386.792280 trunc from 0x954302 to 0x849595 (0x10ad6c bytes) 000081[1] 1772470386.839855 mapread 0x627000 thru 0x62b6ad (0x46ae bytes) 000082[0] 1772470386.922802 write 0x028000 thru 0x029558 (0x1559 bytes) 000083[0] 1772470386.986739 read 0x535000 thru 0x54221a (0xd21b bytes) 000084[1] 1772470387.006823 write_OD 0x07e000 thru 0x087fff (0xa000 bytes) 000086[1] 1772470387.111981 read_OD 0x58d000 thru 0x591fff (0x5000 bytes) 000087[1] 1772470387.124208 mapwrite 0x22d000 thru 0x22dfdb (0xfdc bytes) 000088[0] 1772470387.242012 trunc from 0x849596 to 0x50e482 (0x33b113 bytes) 000089[1] 1772470387.360730 trunc from 0x50e483 to 0x2a8696 (0x265dec bytes) 000090[1] 1772470387.403344 mapwrite 0x5f3000 thru 0x5fb897 (0x8898 bytes) 000091[1] 1772470387.540346 trunc from 0x5fb898 to 0x7b34fc (0x1b7c65 bytes) 000096[0] 1772470387.668827 trunc from 0x7b34fd to 0x1a8dc6 (0x60a736 bytes) 000098[1] 1772470387.757371 mapread 0x01e000 thru 0x01fa3a (0x1a3b bytes) 000103[0] 1772470387.823642 trunc from 0x1a8dc7 to 0x3bef00 (0x21613a bytes) 000104[1] 1772470387.916455 read_OD 0x2eb000 thru 0x2f2fff (0x8000 bytes) 000108[0] 1772470387.966078 mapwrite 0x29d000 thru 0x2a9af1 (0xcaf2 bytes) 000109[0] 1772470388.146627 mapread 0x3b5000 thru 0x3b87d6 (0x37d7 bytes) 000110[0] 1772470388.280750 write 0x9cb000 thru 0x9cbdd6 (0xdd7 bytes) 000111[0] 1772470388.346923 read 0x2da000 thru 0x2e06b3 (0x66b4 bytes) 000112[0] 1772470388.390577 trunc from 0x9cbdd7 to 0x631388 (0x39aa4e bytes) 000113[1] 1772470388.455494 trunc from 0x631389 to 0x46cb2a (0x1c485e bytes) 000114[0] 1772470388.569796 write 0x80c000 thru 0x810bb4 (0x4bb5 bytes) 000115[1] 1772470388.620756 mapwrite 0x978000 thru 0x984ce7 (0xcce8 bytes) 000121[0] 1772470389.351635 write 0x4c6000 thru 0x4c69ea (0x9eb bytes) 000123[0] 1772470389.421790 read 0x828000 thru 0x835ea9 (0xdeaa bytes) 000124[0] 1772470389.440768 trunc from 0x984ce8 to 0x350c55 (0x634092 bytes) 000125[0] 1772470389.548861 write 0x460000 thru 0x46cb21 (0xcb22 bytes) 000126[0] 1772470389.555475 mapread 0x1ca000 thru 0x1d460d (0xa60e bytes) 000127[1] 1772470389.599338 write_OD 0x9c9000 thru 0x9cefff (0x6000 bytes) 000128[0] 1772470389.713152 mapwrite 0x099000 thru 0x0a15ac (0x85ad bytes) 000129[1] 1772470389.951058 mapread 0x16d000 thru 0x16e90e (0x190f bytes) 000130[1] 1772470390.033094 mapwrite 0x967000 thru 0x96e9f4 (0x79f5 bytes) 000133[0] 1772470390.228032 trunc from 0x9cf000 to 0x5e1b57 (0x3ed4a8 bytes) 000135[0] 1772470390.268461 write 0x5d0000 thru 0x5d1ce7 (0x1ce8 bytes) 000137[0] 1772470390.280927 write 0x11e000 thru 0x125ef9 (0x7efa bytes) 000140[0] 1772470390.375370 mapread 0x1e3000 thru 0x1ec5e1 (0x95e2 bytes) 000141[1] 1772470390.444748 mapwrite 0x39c000 thru 0x3a0f2a (0x4f2b bytes) 000143[0] 1772470390.561976 mapread 0x18a000 thru 0x18ccf4 (0x2cf5 bytes) 000149[0] 1772470390.608656 mapwrite 0x508000 thru 0x50aa5c (0x2a5d bytes) 000150[1] 1772470390.850416 mapwrite 0x58b000 thru 0x58b78d (0x78e bytes) 000151[1] 1772470390.947253 write_OD 0x159000 thru 0x15dfff (0x5000 bytes) 000155[0] 1772470391.047910 mapwrite 0x117000 thru 0x117528 (0x529 bytes) 000157[0] 1772470391.254099 read 0x59f000 thru 0x5a1c55 (0x2c56 bytes) 000160[1] 1772470391.324610 trunc from 0x5e1b58 to 0x14b751 (0x496406 bytes) 000161[0] 1772470391.473904 write 0x22f000 thru 0x231183 (0x2184 bytes) 000162[1] 1772470391.531501 mapread 0x1f8000 thru 0x1fb7aa (0x37ab bytes) 000163[0] 1772470391.613189 write 0x966000 thru 0x966ac5 (0xac6 bytes) 000165[0] 1772470391.684076 write 0x0fb000 thru 0x108d10 (0xdd11 bytes) 000166[1] 1772470391.708737 mapread 0x35a000 thru 0x36790b (0xd90c bytes) 000170[1] 1772470391.849406 write_OD 0x031000 thru 0x03efff (0xe000 bytes) 000171[1] 1772470391.963271 read_OD 0x2cd000 thru 0x2d9fff (0xd000 bytes) 000172[1] 1772470391.981669 write_OD 0x6a3000 thru 0x6b1fff (0xf000 bytes) 000173[1] 1772470392.056216 trunc from 0x966ac6 to 0x0307e8 (0x9362dd bytes) 000180[1] 1772470392.154047 write_OD 0x0cb000 thru 0x0d2fff (0x8000 bytes) 000181[1] 1772470392.209148 read_OD 0x053000 thru 0x059fff (0x7000 bytes) 000183[0] 1772470392.225437 trunc from 0x0d3000 to 0x8c3d46 (0x7f0d47 bytes) 000185[0] 1772470392.309437 read 0x135000 thru 0x13a1be (0x51bf bytes) 000187[0] 1772470392.398056 read 0x7e0000 thru 0x7ec1f0 (0xc1f1 bytes) 000188[0] 1772470392.423280 write 0x365000 thru 0x36f342 (0xa343 bytes) 000192[0] 1772470392.517411 write 0x329000 thru 0x33734e (0xe34f bytes) 000193[0] 1772470392.553682 write 0x999000 thru 0x99eca9 (0x5caa bytes) 000195[0] 1772470392.605871 mapwrite 0x23d000 thru 0x2424b3 (0x54b4 bytes) 000196[1] 1772470392.828253 mapread 0x1cf000 thru 0x1dc415 (0xd416 bytes) 000198[0] 1772470392.996189 mapread 0x41d000 thru 0x42231c (0x531d bytes) 000200[0] 1772470393.040555 mapwrite 0x3da000 thru 0x3dd8fb (0x38fc bytes) 000204[1] 1772470393.208446 mapwrite 0x1da000 thru 0x1dab73 (0xb74 bytes) 000205[0] 1772470393.348035 trunc from 0x99ecaa to 0x350e20 (0x64de89 bytes) 000207[0] 1772470393.408781 trunc from 0x350e21 to 0x62f15e (0x2de33e bytes) 000209[0] 1772470393.486206 write 0x876000 thru 0x884946 (0xe947 bytes) 000211[1] 1772470393.537049 mapwrite 0x87a000 thru 0x886e73 (0xce74 bytes) 000213[0] 1772470393.831816 mapwrite 0x2f8000 thru 0x305012 (0xd013 bytes) truncating to largest ever: 0x96d6d9 000214[1] 1772470394.015737 trunc from 0x886e74 to 0x96d6d8 (0xe6865 bytes) 000215[0] 1772470394.104707 trunc from 0x96d6d9 to 0x76a0b8 (0x203620 bytes) 000216[0] 1772470394.192563 mapwrite 0x5f3000 thru 0x6011be (0xe1bf bytes) 000218[0] 1772470394.360594 mapread 0x39c000 thru 0x3a6242 (0xa243 bytes) 000220[1] 1772470394.451877 read_OD 0x39f000 thru 0x3a4fff (0x6000 bytes) 000221[0] 1772470394.538596 read 0x271000 thru 0x280ce8 (0xfce9 bytes) 000223[1] 1772470394.584484 trunc from 0x76a0b9 to 0x063346 (0x706d72 bytes) 000226[1] 1772470394.759434 mapread 0x03f000 thru 0x03f736 (0x737 bytes) 000231[0] 1772470394.867486 read 0x036000 thru 0x03c2ed (0x62ee bytes) 000233[0] 1772470394.958346 mapread 0x031000 thru 0x032a40 (0x1a41 bytes) 000234[0] 1772470394.959230 mapread 0x048000 thru 0x056ae5 (0xeae6 bytes) 000238[0] 1772470394.966034 mapread 0x025000 thru 0x02ccf5 (0x7cf6 bytes) 000240[0] 1772470394.968494 write 0x840000 thru 0x84098e (0x98f bytes) 000242[0] 1772470395.058816 read 0x3ff000 thru 0x409462 (0xa463 bytes) 000245[1] 1772470395.138444 mapread 0x6aa000 thru 0x6af422 (0x5423 bytes) truncating to largest ever: 0x98c442 000249[1] 1772470395.212318 trunc from 0x84098f to 0x98c441 (0x14bab3 bytes) 000250[1] 1772470395.315614 mapread 0x7bf000 thru 0x7cb2de (0xc2df bytes) 000251[1] 1772470395.358049 trunc from 0x98c442 to 0x60ce0a (0x37f637 bytes) 000253[1] 1772470395.421609 write_OD 0x914000 thru 0x920fff (0xd000 bytes) 000254[0] 1772470395.543929 mapread 0x73e000 thru 0x742840 (0x4841 bytes) 000255[1] 1772470395.584892 mapwrite 0x570000 thru 0x57dcff (0xdd00 bytes) 000256[1] 1772470395.737506 mapread 0x7f0000 thru 0x7f5738 (0x5739 bytes) 000258[1] 1772470395.837193 write_OD 0x218000 thru 0x219fff (0x2000 bytes) 000262[0] 1772470395.947468 mapread 0x484000 thru 0x487b06 (0x3b07 bytes) 000265[1] 1772470396.008270 read_OD 0x09a000 thru 0x0a5fff (0xc000 bytes) 000266[1] 1772470396.022399 mapread 0x3b5000 thru 0x3ba6ef (0x56f0 bytes) 000270[1] 1772470396.050576 write_OD 0x18b000 thru 0x190fff (0x6000 bytes) 000271[0] 1772470396.154724 mapread 0x637000 thru 0x641448 (0xa449 bytes) 000275[0] 1772470396.203635 mapwrite 0x12c000 thru 0x13206b (0x606c bytes) 000277[0] 1772470396.400093 mapread 0x25c000 thru 0x25df3f (0x1f40 bytes) 000279[0] 1772470396.445849 write 0x010000 thru 0x01d5f1 (0xd5f2 bytes) 000280[0] 1772470396.515896 mapwrite 0x942000 thru 0x942506 (0x507 bytes) 000283[1] 1772470396.675756 mapread 0x4e8000 thru 0x4f111f (0x9120 bytes) 000285[0] 1772470396.735499 mapwrite 0x5c8000 thru 0x5cc3ff (0x4400 bytes) 000286[1] 1772470396.839360 trunc from 0x942507 to 0x098270 (0x8aa296 bytes) 000287[1] 1772470396.894452 read_OD 0x087000 thru 0x08efff (0x8000 bytes) 000288[1] 1772470396.962330 trunc from 0x098271 to 0x22afd1 (0x192d61 bytes) 000291[1] 1772470397.043847 mapwrite 0x5e8000 thru 0x5ebf48 (0x3f49 bytes) 000292[0] 1772470397.308190 mapwrite 0x052000 thru 0x053ae3 (0x1ae4 bytes) 000293[0] 1772470397.505021 trunc from 0x5ebf49 to 0x4f7747 (0xf4801 bytes) 000297[0] 1772470397.630627 mapread 0x414000 thru 0x41c84f (0x8850 bytes) 000300[1] 1772470397.742148 mapwrite 0x990000 thru 0x990b18 (0xb19 bytes) 000302[0] 1772470398.258355 write 0x1f0000 thru 0x1f839b (0x839c bytes) 000303[0] 1772470398.304463 mapread 0x7d9000 thru 0x7e2d41 (0x9d42 bytes) 000304[0] 1772470398.362968 mapwrite 0x7da000 thru 0x7db0c7 (0x10c8 bytes) 000307[1] 1772470398.531302 read_OD 0x528000 thru 0x534fff (0xd000 bytes) 000309[0] 1772470398.595474 trunc from 0x990b19 to 0x2a8cfe (0x6e7e1a bytes) 000310[1] 1772470398.642356 mapread 0x245000 thru 0x249a05 (0x4a06 bytes) 000311[1] 1772470398.702403 mapwrite 0x502000 thru 0x50c502 (0xa503 bytes) 000312[1] 1772470398.863992 trunc from 0x50c503 to 0x5953c3 (0x88ec1 bytes) 000314[1] 1772470398.889969 read_OD 0x3b3000 thru 0x3bbfff (0x9000 bytes) 000315[0] 1772470398.926068 read 0x38a000 thru 0x395fe9 (0xbfea bytes) 000317[1] 1772470399.044475 trunc from 0x5953c4 to 0x881d08 (0x2ec945 bytes) 000319[0] 1772470399.137841 mapread 0x26c000 thru 0x26f104 (0x3105 bytes) 000320[0] 1772470399.273173 mapwrite 0x3c0000 thru 0x3c7c5a (0x7c5b bytes) 000321[1] 1772470399.499510 trunc from 0x881d09 to 0x12d0ca (0x754c3e bytes) 000322[1] 1772470399.546669 mapread 0x0b6000 thru 0x0b8e37 (0x2e38 bytes) 000324[0] 1772470399.676847 mapread 0x0d6000 thru 0x0d6951 (0x952 bytes) 000326[0] 1772470399.747605 read 0x0e1000 thru 0x0ed933 (0xc934 bytes) 000328[0] 1772470399.813945 trunc from 0x12d0cb to 0x6e2da3 (0x5b5cd9 bytes) 000329[0] 1772470399.916481 read 0x1be000 thru 0x1be4b9 (0x4ba bytes) 000330[0] 1772470399.970020 mapread 0x60f000 thru 0x61a1fc (0xb1fd bytes) 000332[0] 1772470400.027647 mapread 0x3f1000 thru 0x3f108c (0x8d bytes) 000334[1] 1772470400.079784 write_OD 0x597000 thru 0x598fff (0x2000 bytes) 000335[0] 1772470400.159306 write 0x7ab000 thru 0x7b6a21 (0xba22 bytes) 000336[0] 1772470400.193480 write 0x3e6000 thru 0x3f42a7 (0xe2a8 bytes) 000337[1] 1772470400.205105 read_OD 0x6f6000 thru 0x6fefff (0x9000 bytes) 000339[0] 1772470400.290887 mapread 0x1db000 thru 0x1e94c9 (0xe4ca bytes) 000342[1] 1772470400.371285 write_OD 0x065000 thru 0x070fff (0xc000 bytes) 000345[0] 1772470400.476402 read 0x554000 thru 0x562bce (0xebcf bytes) 000346[0] 1772470400.486510 read 0x3fd000 thru 0x408c1e (0xbc1f bytes) 000347[0] 1772470400.498704 read 0x58c000 thru 0x590461 (0x4462 bytes) 000351[0] 1772470400.509505 mapwrite 0x723000 thru 0x728b67 (0x5b68 bytes) 000353[0] 1772470400.599608 mapread 0x147000 thru 0x14e5df (0x75e0 bytes) 000354[1] 1772470400.661459 trunc from 0x7b6a22 to 0x394128 (0x4228f9 bytes) 000355[0] 1772470400.743681 trunc from 0x394129 to 0x506ddb (0x172cb3 bytes) 000356[0] 1772470400.825720 write 0x147000 thru 0x1537d9 (0xc7da bytes) 000357[0] 1772470400.864804 mapwrite 0x01a000 thru 0x02439b (0xa39c bytes) 000359[0] 1772470401.016887 trunc from 0x506ddc to 0x2023f5 (0x3049e6 bytes) 000360[1] 1772470401.049232 read_OD 0x1d1000 thru 0x1dafff (0xa000 bytes) 000364[1] 1772470401.100588 mapwrite 0x75b000 thru 0x768a78 (0xda79 bytes) 000366[1] 1772470401.265765 mapwrite 0x9f7000 thru 0x9ffcbe (0x8cbf bytes) 000367[0] 1772470401.494716 mapread 0x90b000 thru 0x90bbd0 (0xbd1 bytes) 000369[0] 1772470401.566373 mapwrite 0x0a9000 thru 0x0ab540 (0x2541 bytes) 000372[0] 1772470401.843675 mapread 0x3f1000 thru 0x3fffba (0xefbb bytes) 000373[1] 1772470401.922185 read_OD 0x7d5000 thru 0x7e0fff (0xc000 bytes) 000376[0] 1772470401.974338 write 0x401000 thru 0x4069b6 (0x59b7 bytes) 000378[0] 1772470402.097429 read 0x849000 thru 0x849ac9 (0xaca bytes) 000379[0] 1772470402.145474 mapread 0x84c000 thru 0x84e3ae (0x23af bytes) 000380[0] 1772470402.217671 write 0x924000 thru 0x925c5e (0x1c5f bytes) 000384[0] 1772470402.240190 write 0x129000 thru 0x129a21 (0xa22 bytes) 000385[1] 1772470402.254234 mapwrite 0x1df000 thru 0x1ece1c (0xde1d bytes) 000388[1] 1772470402.450112 mapwrite 0x1ca000 thru 0x1cc392 (0x2393 bytes) 000389[0] 1772470402.598011 mapread 0x11d000 thru 0x121f56 (0x4f57 bytes) 000390[0] 1772470402.744296 write 0x2a2000 thru 0x2b01b0 (0xe1b1 bytes) 000391[1] 1772470402.823170 write_OD 0x356000 thru 0x362fff (0xd000 bytes) truncating to largest ever: 0x9add4c 000393[1] 1772470402.988777 trunc from 0x9ffcbf to 0x9add4b (0x51f73 bytes) 000395[0] 1772470403.106021 mapwrite 0x3f1000 thru 0x3f4512 (0x3513 bytes) 000398[1] 1772470403.507196 mapread 0x7ba000 thru 0x7c0e04 (0x6e05 bytes) 000400[0] 1772470403.548852 read 0x0cf000 thru 0x0d4aeb (0x5aec bytes) 000402[0] 1772470403.562926 write 0x360000 thru 0x3664ba (0x64bb bytes) 000403[1] 1772470403.584233 mapread 0x28c000 thru 0x297821 (0xb822 bytes) 000407[1] 1772470403.682811 trunc from 0x9add4c to 0x98a721 (0x2362a bytes) 000408[0] 1772470403.741110 trunc from 0x98a722 to 0x401637 (0x5890ea bytes) 000409[1] 1772470403.769141 mapread 0x017000 thru 0x024e08 (0xde09 bytes) truncating to largest ever: 0x9dbeca 000411[1] 1772470403.845710 trunc from 0x401638 to 0x9dbec9 (0x5da892 bytes) 000412[1] 1772470403.890689 mapwrite 0x34c000 thru 0x34eea8 (0x2ea9 bytes) 000413[1] 1772470404.049349 read_OD 0x94a000 thru 0x958fff (0xf000 bytes) 000416[1] 1772470404.095683 read_OD 0x0f2000 thru 0x0fdfff (0xc000 bytes) 000419[1] 1772470404.116599 read_OD 0x3a3000 thru 0x3a9fff (0x7000 bytes) truncating to largest ever: 0x9e82bf 000423[1] 1772470404.145250 trunc from 0x9dbeca to 0x9e82be (0xc3f5 bytes) 000425[1] 1772470404.317790 mapread 0x03e000 thru 0x03f357 (0x1358 bytes) 000428[1] 1772470404.444487 trunc from 0x9e82bf to 0x0ff8f9 (0x8e89c5 bytes) 000429[0] 1772470404.492301 read 0x00f000 thru 0x01ef2d (0xff2e bytes) 000430[1] 1772470404.590719 write_OD 0x835000 thru 0x838fff (0x4000 bytes) 000432[1] 1772470404.773791 mapwrite 0x960000 thru 0x96940a (0x940b bytes) 000436[0] 1772470405.127647 trunc from 0x96940b to 0x079da8 (0x8ef662 bytes) 000439[0] 1772470405.230058 mapread 0x041000 thru 0x0482c2 (0x72c3 bytes) 000442[1] 1772470405.285480 write_OD 0x7d5000 thru 0x7d8fff (0x4000 bytes) 000443[0] 1772470405.443210 mapwrite 0x891000 thru 0x8931b5 (0x21b6 bytes) 000444[1] 1772470405.653976 trunc from 0x8931b6 to 0x905bea (0x72a35 bytes) 000447[1] 1772470405.813428 mapread 0x0db000 thru 0x0e37be (0x87bf bytes) 000449[1] 1772470405.861265 mapwrite 0x069000 thru 0x074210 (0xb211 bytes) 000450[1] 1772470406.036960 trunc from 0x905beb to 0x8d3172 (0x32a78 bytes) 000454[1] 1772470406.151322 mapwrite 0x019000 thru 0x027a57 (0xea58 bytes) 000455[1] 1772470406.314091 mapwrite 0x3ad000 thru 0x3aeed8 (0x1ed9 bytes) 000456[1] 1772470406.494253 read_OD 0x2c2000 thru 0x2cffff (0xe000 bytes) 000462[1] 1772470406.571163 write_OD 0x293000 thru 0x296fff (0x4000 bytes) 000463[0] 1772470406.716180 write 0x835000 thru 0x836512 (0x1513 bytes) 000464[0] 1772470406.846062 read 0x82c000 thru 0x82e432 (0x2433 bytes) 000471[0] 1772470406.947332 trunc from 0x8d3173 to 0x8aca37 (0x2673b bytes) 000474[1] 1772470407.084484 mapwrite 0x582000 thru 0x58bcb5 (0x9cb6 bytes) 000475[0] 1772470407.267673 mapwrite 0x743000 thru 0x745301 (0x2302 bytes) 000476[0] 1772470407.407111 mapread 0x102000 thru 0x110fd4 (0xefd5 bytes) 000479[0] 1772470407.466858 mapwrite 0x06e000 thru 0x07885e (0xa85f bytes) 000480[0] 1772470407.581499 trunc from 0x8aca38 to 0x891fe1 (0x1aa56 bytes) 000482[0] 1772470407.647150 write 0x48b000 thru 0x497671 (0xc672 bytes) 000483[0] 1772470407.656377 read 0x142000 thru 0x145738 (0x3739 bytes) 000486[0] 1772470407.673567 trunc from 0x891fe2 to 0x2b33a8 (0x5dec39 bytes) 000487[0] 1772470407.729303 mapwrite 0x2a7000 thru 0x2aabce (0x3bcf bytes) 000490[0] 1772470407.896030 mapwrite 0x3f9000 thru 0x3ff6eb (0x66ec bytes) 000493[0] 1772470408.092421 write 0x471000 thru 0x477d21 (0x6d22 bytes) 000495[1] 1772470408.141227 trunc from 0x477d22 to 0x5e1749 (0x169a28 bytes) All operations completed A-OK! PASS 16a (70s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 11:53:34 (1772470414) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.471883 s, 17.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.35976 s, 23.3 MB/s Chance of close/open is 1 in 50 Seed set to 417 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1772470418.679023 mapwrite 0x0af428 thru 0x0b0026 (0xbff bytes) 000002[0] 1772470418.808388 write 0x64a63f thru 0x64ba9e (0x1460 bytes) 000003[0] 1772470418.852563 read 0x0159ee thru 0x016a35 (0x1048 bytes) truncating to largest ever: 0xba5b3 000004[0] 1772470418.895089 trunc from 0x64ba9f to 0x0ba5b2 (0x5914ec bytes) 000006[1] 1772470418.924082 write 0x98f8c6 thru 0x990587 (0xcc2 bytes) 000010[1] 1772470419.078145 read 0x2f5cf8 thru 0x2f7afa (0x1e03 bytes) truncating to largest ever: 0x3f7efc 000011[0] 1772470419.137102 trunc from 0x990588 to 0x3f7efb (0x59868c bytes) 000012[0] 1772470419.211445 mapread 0x08e806 thru 0x08e943 (0x13e bytes) 000014[1] 1772470419.258846 read 0x3221d6 thru 0x323410 (0x123b bytes) 000021[0] 1772470419.455151 write 0x944e38 thru 0x9457c0 (0x989 bytes) truncating to largest ever: 0x7771f2 000023[0] 1772470419.521919 trunc from 0x9457c1 to 0x7771f1 (0x1ce5cf bytes) 000024[0] 1772470419.554818 trunc from 0x7771f2 to 0x06f3df (0x707e12 bytes) 000029[1] 1772470419.688163 trunc from 0x06f3e0 to 0x2a7f7c (0x238b9d bytes) 000030[0] 1772470419.869238 trunc from 0x2a7f7d to 0x32148c (0x79510 bytes) truncating to largest ever: 0x8cb495 000032[0] 1772470420.164117 trunc from 0x32148d to 0x8cb494 (0x5aa008 bytes) 000033[0] 1772470420.253628 mapwrite 0x481b1f thru 0x48284f (0xd31 bytes) 000039[1] 1772470420.493880 read 0x6d5e13 thru 0x6d6705 (0x8f3 bytes) 000040[0] 1772470420.518460 read 0x004850 thru 0x00650c (0x1cbd bytes) 000041[1] 1772470420.530785 trunc from 0x8cb495 to 0x4d84dc (0x3f2fb8 bytes) 000045[1] 1772470420.698902 mapread 0x0f2afb thru 0x0f3690 (0xb96 bytes) 000052[0] 1772470420.877445 mapread 0x1c5a2a thru 0x1c650b (0xae2 bytes) 000053[1] 1772470420.941493 read 0x3a8fd4 thru 0x3aaf30 (0x1f5d bytes) 000056[0] 1772470420.977790 mapwrite 0x053cee thru 0x055811 (0x1b24 bytes) 000057[0] 1772470421.221826 read 0x0c2669 thru 0x0c2c55 (0x5ed bytes) truncating to largest ever: 0x91be6b 000058[0] 1772470421.253062 trunc from 0x4d84dd to 0x91be6a (0x44398e bytes) 000060[0] 1772470421.333795 mapread 0x6000e4 thru 0x6016aa (0x15c7 bytes) 000061[1] 1772470421.371784 mapread 0x61019c thru 0x611e51 (0x1cb6 bytes) 000062[1] 1772470421.443000 mapread 0x75a665 thru 0x75c2ea (0x1c86 bytes) 000063[0] 1772470421.472392 write 0x987890 thru 0x989332 (0x1aa3 bytes) 000064[0] 1772470421.490194 read 0x14d0bf thru 0x14e532 (0x1474 bytes) 000065[0] 1772470421.497427 trunc from 0x989333 to 0x6d5926 (0x2b3a0c bytes) 000067[0] 1772470421.573241 mapread 0x12c65f thru 0x12d08e (0xa30 bytes) truncating to largest ever: 0x99f0a4 000069[0] 1772470421.634557 trunc from 0x6d5927 to 0x99f0a3 (0x2c977d bytes) 000071[0] 1772470421.702099 write 0x4e31d2 thru 0x4e5011 (0x1e40 bytes) 000072[0] 1772470421.748446 trunc from 0x99f0a4 to 0x189e27 (0x81527c bytes) 000073[1] 1772470421.801202 read 0x0a308b thru 0x0a4a5d (0x19d3 bytes) 000074[0] 1772470421.880435 trunc from 0x189e28 to 0x666dfb (0x4dcfd4 bytes) 000075[1] 1772470421.946000 trunc from 0x666dfc to 0x97b17f (0x314384 bytes) 000076[1] 1772470422.005626 trunc from 0x97b180 to 0x625949 (0x355836 bytes) 000080[1] 1772470422.028988 mapread 0x2985a4 thru 0x299197 (0xbf4 bytes) 000083[1] 1772470422.221604 mapwrite 0x0fc7b9 thru 0x0fd3b7 (0xbff bytes) 000084[1] 1772470422.286543 mapwrite 0x52067f thru 0x52108d (0xa0f bytes) 000086[0] 1772470422.436280 write 0x291ad4 thru 0x29314f (0x167c bytes) 000088[1] 1772470422.555574 mapwrite 0x789faa thru 0x78a127 (0x17e bytes) 000092[1] 1772470422.934624 mapread 0x0559f8 thru 0x057013 (0x161c bytes) 000093[1] 1772470422.969510 trunc from 0x78a128 to 0x65bae2 (0x12e645 bytes) 000096[1] 1772470423.069125 write 0x4e405b thru 0x4e48f1 (0x897 bytes) 000098[0] 1772470423.098367 write 0x9141e2 thru 0x915547 (0x1366 bytes) 000101[1] 1772470423.178384 read 0x264018 thru 0x264e1e (0xe07 bytes) 000102[1] 1772470423.233432 trunc from 0x915548 to 0x4fe353 (0x4171f4 bytes) 000103[0] 1772470423.291314 read 0x4a2eeb thru 0x4a4680 (0x1796 bytes) 000105[0] 1772470423.334695 write 0x091f51 thru 0x0931b0 (0x1260 bytes) 000108[0] 1772470423.436462 mapread 0x2d01de thru 0x2d1718 (0x153b bytes) 000109[0] 1772470423.474417 read 0x1a30a0 thru 0x1a4c8f (0x1bf0 bytes) 000114[1] 1772470423.543151 write 0x5380e2 thru 0x539d83 (0x1ca2 bytes) 000115[0] 1772470423.591906 read 0x029fa5 thru 0x02a75a (0x7b6 bytes) 000118[1] 1772470423.672705 trunc from 0x539d84 to 0x919147 (0x3df3c4 bytes) 000120[1] 1772470423.795760 read 0x61df0f thru 0x61e67e (0x770 bytes) 000122[1] 1772470423.812375 write 0x620c47 thru 0x620dc5 (0x17f bytes) 000123[0] 1772470423.850152 mapwrite 0x4aa4d1 thru 0x4abf39 (0x1a69 bytes) 000126[0] 1772470424.063742 write 0x1a31b9 thru 0x1a31d4 (0x1c bytes) 000127[0] 1772470424.100435 mapwrite 0x257c30 thru 0x257d67 (0x138 bytes) 000134[1] 1772470424.334079 mapread 0x738623 thru 0x739ad6 (0x14b4 bytes) 000135[0] 1772470424.400064 read 0x79ca65 thru 0x79e809 (0x1da5 bytes) 000138[0] 1772470424.445151 mapread 0x358716 thru 0x3592e6 (0xbd1 bytes) 000139[0] 1772470424.468456 trunc from 0x919148 to 0x1707fd (0x7a894a bytes) 000140[0] 1772470424.566158 read 0x047105 thru 0x048334 (0x1230 bytes) 000142[0] 1772470424.688188 read 0x133de2 thru 0x13502c (0x124b bytes) 000143[0] 1772470424.690417 write 0x6d11b0 thru 0x6d2acd (0x191e bytes) 000144[1] 1772470424.702232 mapread 0x4e3bf4 thru 0x4e430b (0x718 bytes) 000145[0] 1772470424.776969 mapread 0x0ff4a5 thru 0x100db8 (0x1914 bytes) 000148[0] 1772470424.852011 read 0x389d17 thru 0x38b1ac (0x1496 bytes) 000149[1] 1772470424.940752 trunc from 0x6d2ace to 0x5c8c87 (0x109e46 bytes) 000153[1] 1772470425.041182 trunc from 0x5c8c88 to 0x06e238 (0x55aa4f bytes) 000155[0] 1772470425.058058 write 0x631ab7 thru 0x633a95 (0x1fdf bytes) 000156[1] 1772470425.140788 mapread 0x16129f thru 0x1626cf (0x1431 bytes) 000158[1] 1772470425.317620 write 0x453c6e thru 0x453f17 (0x2aa bytes) 000160[1] 1772470425.361570 read 0x2bd890 thru 0x2be60b (0xd7c bytes) 000162[0] 1772470425.429598 mapread 0x02c301 thru 0x02e1a2 (0x1ea2 bytes) 000164[1] 1772470425.473217 write 0x8db882 thru 0x8dc135 (0x8b4 bytes) 000166[0] 1772470425.546255 read 0x881f2d thru 0x883b6f (0x1c43 bytes) 000167[1] 1772470425.591072 read 0x5c856f thru 0x5c8618 (0xaa bytes) 000169[1] 1772470425.622467 mapwrite 0x3ce767 thru 0x3cf67c (0xf16 bytes) 000170[0] 1772470425.749488 read 0x836273 thru 0x836f9c (0xd2a bytes) 000173[0] 1772470425.852743 read 0x8ae991 thru 0x8ae99d (0xd bytes) 000174[0] 1772470425.892971 trunc from 0x8dc136 to 0x622b00 (0x2b9635 bytes) 000176[0] 1772470426.062242 mapread 0x2f2d21 thru 0x2f2ed1 (0x1b1 bytes) 000177[1] 1772470426.141129 trunc from 0x622b01 to 0x1f1112 (0x4319ee bytes) 000178[1] 1772470426.326013 mapwrite 0x1e7e0d thru 0x1e853d (0x731 bytes) 000180[0] 1772470426.533966 mapwrite 0x0ef8cb thru 0x0efa65 (0x19b bytes) 000181[0] 1772470426.746417 mapread 0x006ea0 thru 0x0081f3 (0x1354 bytes) 000182[0] 1772470426.854169 mapread 0x02550b thru 0x026604 (0x10fa bytes) 000183[0] 1772470426.950104 mapread 0x195135 thru 0x195b0a (0x9d6 bytes) 000185[1] 1772470427.116282 mapread 0x10f110 thru 0x110fbc (0x1ead bytes) 000188[1] 1772470427.286235 trunc from 0x1f1113 to 0x270043 (0x7ef31 bytes) 000193[1] 1772470427.415242 mapwrite 0x938e4d thru 0x939818 (0x9cc bytes) 000196[1] 1772470427.705080 read 0x6fa426 thru 0x6faef4 (0xacf bytes) 000197[1] 1772470427.739021 mapread 0x7e4017 thru 0x7e4155 (0x13f bytes) 000199[1] 1772470427.784624 write 0x3f2e8b thru 0x3f4668 (0x17de bytes) 000202[0] 1772470427.956278 mapwrite 0x2e44c1 thru 0x2e49cd (0x50d bytes) 000203[1] 1772470428.084370 write 0x9ba072 thru 0x9bb0f5 (0x1084 bytes) 000204[0] 1772470428.137257 read 0x6dec92 thru 0x6e0aba (0x1e29 bytes) 000206[1] 1772470428.198059 mapwrite 0x17b64d thru 0x17c0ec (0xaa0 bytes) 000208[1] 1772470428.353181 mapread 0x282369 thru 0x2833b8 (0x1050 bytes) 000212[1] 1772470428.507123 write 0x724a8d thru 0x726874 (0x1de8 bytes) 000213[1] 1772470428.569973 mapwrite 0x747be8 thru 0x74880c (0xc25 bytes) 000216[1] 1772470428.641441 read 0x8ff0f2 thru 0x900cfa (0x1c09 bytes) 000221[0] 1772470428.750412 mapwrite 0x56ce68 thru 0x56d470 (0x609 bytes) 000222[0] 1772470428.855158 mapread 0x86d71d thru 0x86d9bb (0x29f bytes) 000223[1] 1772470428.908468 write 0x990591 thru 0x99059b (0xb bytes) 000224[1] 1772470428.974253 read 0x497283 thru 0x497fdb (0xd59 bytes) 000225[1] 1772470428.996499 mapwrite 0x61ab70 thru 0x61c1b8 (0x1649 bytes) 000226[0] 1772470429.096404 trunc from 0x9bb0f6 to 0x61a614 (0x3a0ae1 bytes) 000229[1] 1772470429.200499 trunc from 0x61a615 to 0x23865e (0x3e1fb6 bytes) 000230[1] 1772470429.294587 write 0x4cb377 thru 0x4cc6b6 (0x1340 bytes) 000231[1] 1772470429.332633 write 0x7c3062 thru 0x7c42d3 (0x1272 bytes) 000232[1] 1772470429.380418 mapwrite 0x23790b thru 0x237bf6 (0x2ec bytes) 000233[0] 1772470429.461312 trunc from 0x7c42d4 to 0x89a6ad (0xd63da bytes) 000234[1] 1772470429.554063 write 0x39c755 thru 0x39d48c (0xd38 bytes) 000235[0] 1772470429.606484 read 0x3bdf06 thru 0x3bf6df (0x17da bytes) 000239[0] 1772470429.715469 write 0x5c7b71 thru 0x5c9679 (0x1b09 bytes) 000241[1] 1772470429.779104 read 0x269ea7 thru 0x26a0a8 (0x202 bytes) 000242[0] 1772470429.807435 write 0x931318 thru 0x93213a (0xe23 bytes) 000244[1] 1772470429.888096 mapread 0x18f43a thru 0x190aff (0x16c6 bytes) 000246[1] 1772470429.984235 mapwrite 0x090e3d thru 0x091697 (0x85b bytes) 000247[1] 1772470430.097967 trunc from 0x93213b to 0x22a184 (0x707fb6 bytes) 000249[1] 1772470430.233400 mapread 0x193c15 thru 0x195565 (0x1951 bytes) 000252[1] 1772470430.284309 trunc from 0x22a185 to 0x245f81 (0x1bdfd bytes) 000253[0] 1772470430.439844 write 0x801481 thru 0x802ca0 (0x1820 bytes) 000254[1] 1772470430.480098 read 0x4ac145 thru 0x4aca9b (0x957 bytes) 000256[0] 1772470430.556479 trunc from 0x802ca1 to 0x78492f (0x7e371 bytes) 000257[0] 1772470430.596412 trunc from 0x784930 to 0x471aa7 (0x312e88 bytes) 000260[0] 1772470430.648774 mapwrite 0x42c49a thru 0x42c519 (0x80 bytes) 000261[0] 1772470430.752750 trunc from 0x471aa8 to 0x872591 (0x400aea bytes) 000262[1] 1772470430.799605 write 0x1caa65 thru 0x1cb656 (0xbf2 bytes) 000268[1] 1772470431.026225 write 0x9e6cdb thru 0x9e7af2 (0xe18 bytes) 000269[1] 1772470431.045312 read 0x25f6a4 thru 0x25f6e1 (0x3e bytes) 000270[0] 1772470431.093057 trunc from 0x9e7af3 to 0x241782 (0x7a6370 bytes) 000271[0] 1772470431.189349 mapwrite 0x2556c2 thru 0x257168 (0x1aa7 bytes) 000272[1] 1772470431.300467 write 0x290d35 thru 0x292917 (0x1be3 bytes) 000274[1] 1772470431.385122 mapwrite 0x2c13b7 thru 0x2c2eb7 (0x1b01 bytes) 000275[0] 1772470431.490320 write 0x8b33a0 thru 0x8b3a24 (0x685 bytes) 000277[1] 1772470431.542037 mapwrite 0x597c59 thru 0x599b91 (0x1f39 bytes) 000278[1] 1772470431.734805 trunc from 0x8b3a25 to 0x603fda (0x2afa4a bytes) 000279[1] 1772470431.787476 read 0x209ea7 thru 0x20b9ba (0x1b14 bytes) 000280[0] 1772470431.795314 read 0x44ece9 thru 0x450b1d (0x1e35 bytes) 000283[0] 1772470431.825621 read 0x0e6a98 thru 0x0e7100 (0x669 bytes) 000284[1] 1772470431.855944 mapread 0x146b66 thru 0x147f3c (0x13d7 bytes) 000285[0] 1772470431.874169 trunc from 0x603fdb to 0x341b4a (0x2c2490 bytes) 000286[0] 1772470431.924057 mapwrite 0x169cd9 thru 0x16ae98 (0x11c0 bytes) 000287[1] 1772470432.021566 trunc from 0x341b4b to 0x6315bf (0x2efa75 bytes) 000288[0] 1772470432.078724 mapread 0x603ca7 thru 0x6047b1 (0xb0b bytes) 000289[1] 1772470432.141179 read 0x417207 thru 0x418281 (0x107b bytes) 000292[1] 1772470432.170089 mapwrite 0x9bd707 thru 0x9bd924 (0x21e bytes) 000294[1] 1772470432.348703 read 0x95ef0c thru 0x95f7bc (0x8b1 bytes) 000296[0] 1772470432.388500 write 0x3539eb thru 0x35487a (0xe90 bytes) 000299[0] 1772470432.529241 mapwrite 0x2777e4 thru 0x2794ce (0x1ceb bytes) 000300[0] 1772470432.628927 write 0x64bfa0 thru 0x64c8e0 (0x941 bytes) 000301[1] 1772470432.657717 read 0x27ea02 thru 0x27f081 (0x680 bytes) 000302[1] 1772470432.703740 mapwrite 0x336fc0 thru 0x338544 (0x1585 bytes) 000303[1] 1772470432.956049 trunc from 0x9bd925 to 0x692abc (0x32ae68 bytes) 000304[0] 1772470433.009453 mapwrite 0x20c517 thru 0x20e506 (0x1ff0 bytes) 000305[1] 1772470433.177985 mapwrite 0x2ec3f3 thru 0x2ed8bd (0x14cb bytes) 000306[0] 1772470433.385727 mapwrite 0x1d282f thru 0x1d3f05 (0x16d7 bytes) 000307[1] 1772470433.589824 mapwrite 0x36798e thru 0x3682ff (0x972 bytes) 000308[0] 1772470433.775842 mapwrite 0x491d80 thru 0x4939ff (0x1c80 bytes) 000309[0] 1772470433.919101 write 0x66b8fb thru 0x66d7e3 (0x1ee9 bytes) 000310[0] 1772470433.945594 mapread 0x4b38ea thru 0x4b3d7d (0x494 bytes) 000311[1] 1772470433.980866 trunc from 0x692abd to 0x0cd637 (0x5c5485 bytes) 000313[1] 1772470434.063895 mapwrite 0x36fab5 thru 0x371237 (0x1783 bytes) 000314[1] 1772470434.165243 read 0x350745 thru 0x352309 (0x1bc5 bytes) 000315[0] 1772470434.219269 read 0x07a001 thru 0x07b6b4 (0x16b4 bytes) 000316[1] 1772470434.241285 trunc from 0x371238 to 0x807aa8 (0x496871 bytes) 000318[1] 1772470434.296162 mapread 0x64f8c7 thru 0x651060 (0x179a bytes) 000319[0] 1772470434.380416 trunc from 0x807aa9 to 0x977c1d (0x170175 bytes) 000323[0] 1772470434.492226 mapwrite 0x2700b5 thru 0x2715b5 (0x1501 bytes) 000324[0] 1772470434.577911 mapwrite 0x46ff42 thru 0x470994 (0xa53 bytes) 000325[1] 1772470434.670038 write 0x34e6a5 thru 0x350288 (0x1be4 bytes) 000326[1] 1772470434.749407 trunc from 0x977c1e to 0x110c83 (0x866f9a bytes) 000329[0] 1772470434.876699 write 0x44f47b thru 0x44fe80 (0xa06 bytes) 000330[1] 1772470434.910859 write 0x708859 thru 0x7089a4 (0x14c bytes) 000331[1] 1772470434.932978 mapwrite 0x59cc7f thru 0x59e6b8 (0x1a3a bytes) 000332[1] 1772470434.996181 mapread 0x670a93 thru 0x67100d (0x57b bytes) 000335[1] 1772470435.011715 read 0x2c57b2 thru 0x2c6499 (0xce8 bytes) 000336[0] 1772470435.024952 mapread 0x008bb4 thru 0x009849 (0xc96 bytes) 000337[0] 1772470435.087058 mapread 0x40908e thru 0x40946d (0x3e0 bytes) 000338[1] 1772470435.129348 write 0x027378 thru 0x0279e8 (0x671 bytes) 000339[0] 1772470435.174374 mapread 0x1efd57 thru 0x1f1751 (0x19fb bytes) 000342[0] 1772470435.234174 mapread 0x6caabb thru 0x6cc721 (0x1c67 bytes) 000346[0] 1772470435.272087 mapwrite 0x7cd424 thru 0x7cf1d9 (0x1db6 bytes) 000348[1] 1772470435.448328 read 0x2ee5b7 thru 0x2efdd9 (0x1823 bytes) 000350[1] 1772470435.512138 read 0x18e99a thru 0x18fe9a (0x1501 bytes) 000353[0] 1772470435.590906 mapwrite 0x42764d thru 0x427dfb (0x7af bytes) 000354[0] 1772470435.655381 read 0x12d095 thru 0x12d57e (0x4ea bytes) 000355[0] 1772470435.687041 write 0x5856e3 thru 0x586eec (0x180a bytes) 000357[1] 1772470435.749858 trunc from 0x7cf1da to 0x75b62f (0x73baa bytes) 000360[0] 1772470435.932923 trunc from 0x75b630 to 0x38dd09 (0x3cd926 bytes) 000361[0] 1772470436.017218 mapwrite 0x592af0 thru 0x593f57 (0x1468 bytes) 000362[0] 1772470436.212965 write 0x123aec thru 0x123ef2 (0x407 bytes) 000363[1] 1772470436.243607 mapread 0x4c54e1 thru 0x4c70e9 (0x1c09 bytes) 000366[0] 1772470436.393186 trunc from 0x593f58 to 0x51df49 (0x7600e bytes) 000372[0] 1772470436.631413 mapread 0x475792 thru 0x476c16 (0x1485 bytes) 000375[1] 1772470436.715797 trunc from 0x51df4a to 0x02f34d (0x4eebfc bytes) 000376[1] 1772470436.790867 mapread 0x013408 thru 0x013fdd (0xbd6 bytes) 000377[0] 1772470436.800330 trunc from 0x02f34e to 0x2ec52a (0x2bd1dd bytes) 000378[0] 1772470436.877399 mapwrite 0x39465e thru 0x394a29 (0x3cc bytes) 000379[0] 1772470436.947667 write 0x2eade4 thru 0x2ec99f (0x1bbc bytes) 000380[0] 1772470436.989895 write 0x9a75be thru 0x9a7fa4 (0x9e7 bytes) 000381[1] 1772470437.023055 trunc from 0x9a7fa5 to 0x1339a5 (0x8745ff bytes) 000383[0] 1772470437.175895 trunc from 0x1339a6 to 0x3b55b7 (0x281c12 bytes) 000389[0] 1772470437.346193 mapwrite 0x7237b7 thru 0x723dc2 (0x60c bytes) 000390[0] 1772470437.496576 mapwrite 0x1ff1c3 thru 0x20017d (0xfbb bytes) 000391[1] 1772470437.593358 mapread 0x17b3da thru 0x17b526 (0x14d bytes) 000395[0] 1772470437.768610 read 0x387470 thru 0x387ec4 (0xa55 bytes) 000397[1] 1772470437.781196 write 0x7f3665 thru 0x7f4762 (0x10fe bytes) 000399[0] 1772470437.825078 mapwrite 0x6c910a thru 0x6ca918 (0x180f bytes) 000402[0] 1772470438.003292 read 0x5f82ff thru 0x5f9703 (0x1405 bytes) 000406[0] 1772470438.016653 read 0x2380b2 thru 0x238aa7 (0x9f6 bytes) 000410[1] 1772470438.053071 mapwrite 0x3bfd4d thru 0x3bfe88 (0x13c bytes) 000411[0] 1772470438.186779 mapwrite 0x810a71 thru 0x811cb7 (0x1247 bytes) 000413[0] 1772470438.346754 mapwrite 0x255383 thru 0x2563c1 (0x103f bytes) 000414[0] 1772470438.520398 write 0x910048 thru 0x911460 (0x1419 bytes) 000415[0] 1772470438.548958 mapread 0x10f2e4 thru 0x10fedb (0xbf8 bytes) 000416[1] 1772470438.564354 read 0x8f6804 thru 0x8f6b18 (0x315 bytes) 000417[0] 1772470438.603276 write 0x609f26 thru 0x60b521 (0x15fc bytes) 000419[0] 1772470438.658343 mapwrite 0x0c7356 thru 0x0c8176 (0xe21 bytes) 000420[0] 1772470438.750683 read 0x061216 thru 0x06286b (0x1656 bytes) 000422[0] 1772470438.800431 trunc from 0x911461 to 0x0344d7 (0x8dcf89 bytes) 000425[0] 1772470438.917403 read 0x0293ba thru 0x029c5c (0x8a3 bytes) 000426[1] 1772470439.048336 trunc from 0x0344d8 to 0x8832f9 (0x84ee22 bytes) 000427[0] 1772470439.171250 write 0x276efc thru 0x2788ca (0x19cf bytes) 000428[0] 1772470439.299130 mapread 0x295349 thru 0x2970aa (0x1d62 bytes) 000429[1] 1772470439.347118 mapwrite 0x0cbe64 thru 0x0cd968 (0x1b05 bytes) 000430[1] 1772470439.571241 read 0x3a17fc thru 0x3a2a98 (0x129d bytes) 000431[1] 1772470439.623031 write 0x1f9429 thru 0x1f9524 (0xfc bytes) 000437[1] 1772470439.888228 write 0x4633ba thru 0x464ac9 (0x1710 bytes) 000440[0] 1772470439.978290 trunc from 0x8832fa to 0x13cc95 (0x746664 bytes) 000442[1] 1772470440.168110 mapread 0x03c0e7 thru 0x03cba2 (0xabc bytes) 000445[1] 1772470440.283238 write 0x9f29d9 thru 0x9f3076 (0x69e bytes) 000449[0] 1772470440.383022 mapread 0x14c816 thru 0x14d313 (0xafe bytes) 000451[0] 1772470440.448110 mapwrite 0x3a86cb thru 0x3a9653 (0xf89 bytes) 000454[1] 1772470440.680392 mapread 0x0ed18d thru 0x0ed843 (0x6b7 bytes) 000455[1] 1772470440.743193 mapwrite 0x9ef39f thru 0x9ef961 (0x5c3 bytes) 000457[0] 1772470440.945103 mapwrite 0x8824e2 thru 0x883950 (0x146f bytes) 000458[1] 1772470441.150787 mapread 0x967242 thru 0x96841d (0x11dc bytes) 000459[0] 1772470441.239764 read 0x50b067 thru 0x50bc51 (0xbeb bytes) 000461[0] 1772470441.265029 write 0x9a6f89 thru 0x9a7cc0 (0xd38 bytes) 000462[0] 1772470441.326842 trunc from 0x9f3077 to 0x2b09ec (0x74268a bytes) 000463[1] 1772470441.403969 mapwrite 0x0698f2 thru 0x06ac99 (0x13a8 bytes) 000465[1] 1772470441.525817 mapread 0x234a14 thru 0x2350c1 (0x6ae bytes) 000466[1] 1772470441.588834 trunc from 0x2b09ed to 0x3564dd (0xa5af1 bytes) 000468[0] 1772470441.624899 trunc from 0x3564de to 0x4202b0 (0xc9dd3 bytes) 000469[0] 1772470441.757896 mapread 0x073ff5 thru 0x0757ae (0x17ba bytes) 000471[0] 1772470441.844806 write 0x65ad8c thru 0x65cd61 (0x1fd6 bytes) 000473[1] 1772470441.884894 write 0x072ef9 thru 0x0739ac (0xab4 bytes) 000474[0] 1772470441.998074 mapread 0x0e05ec thru 0x0e1a75 (0x148a bytes) 000475[0] 1772470442.073396 trunc from 0x65cd62 to 0x14dc86 (0x50f0db bytes) 000476[1] 1772470442.227172 read 0x0986b7 thru 0x09a235 (0x1b7f bytes) 000477[1] 1772470442.366066 mapwrite 0x264613 thru 0x264daa (0x798 bytes) 000479[1] 1772470442.702380 trunc from 0x264dab to 0x8a9d37 (0x644f8d bytes) 000482[1] 1772470442.815793 read 0x72ff8c thru 0x731b62 (0x1bd7 bytes) 000488[0] 1772470442.942976 write 0x244220 thru 0x245f77 (0x1d58 bytes) 000490[0] 1772470443.013145 trunc from 0x8a9d38 to 0x6d4ca1 (0x1d5096 bytes) 000492[1] 1772470443.107846 read 0x53516b thru 0x53556a (0x400 bytes) 000493[0] 1772470443.185159 mapwrite 0x969cad thru 0x96b2a4 (0x15f8 bytes) 000494[1] 1772470443.513479 mapread 0x73c9d8 thru 0x73cd66 (0x38f bytes) 000496[0] 1772470443.621148 read 0x73f61f thru 0x73fb1a (0x4fc bytes) 000497[1] 1772470443.637069 mapwrite 0x25f380 thru 0x2601c1 (0xe42 bytes) 000498[0] 1772470443.765105 mapread 0x8d823a thru 0x8d8e4f (0xc16 bytes) 000500[0] 1772470443.827391 trunc from 0x96b2a5 to 0x64f224 (0x31c080 bytes) All operations completed A-OK! PASS 16b (35s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 11:54:09 (1772470449) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 11:54:14 (1772470454) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.432974 s, 19.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.399252 s, 21.0 MB/s ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8e8008cf6800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8e80202cc800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8e8008cf6800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8e8008cf6800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.8197 s, 8.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.1256 s, 33.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.55156 s, 29.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.1882 s, 14.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.84943 s, 11.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.91807 s, 26.8 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 11:55:09 (1772470509) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.20873 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214106 s, 4.9 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 11:55:17 (1772470517) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 11:55:45 (1772470545) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 11:56:15 (1772470575) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (11s) == sanityn test 16i: read after truncate file ============ 11:56:25 (1772470585) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 11:56:33 (1772470593) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.39249 s, 7.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83964 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.83075 s, 6.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50018 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68629 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79605 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.02556 s, 6.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.31711 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37471 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.25575 s, 5.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.42524 s, 4.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.02252 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.38803 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.10333 s, 6.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.33433 s, 6.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.48217 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.29152 s, 5.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.66156 s, 4.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.89332 s, 4.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (58s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 11:57:31 (1772470651) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Mar 2 11:58:25 EST 2026 + fsx 2 done: rc=0 Mon Mar 2 11:58:25 EST 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Mon Mar 2 11:58:26 EST 2026 + fsx 4 done: rc=0 Mon Mar 2 11:58:26 EST 2026 + fsx 5 done: rc=0 Mon Mar 2 11:58:26 EST 2026 PASS 16k (64s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 11:58:35 (1772470715) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 11:58:47 (1772470727) mmap test1: basic mmap operation (PASS, 0.15578s) mmap test2: MAP_PRIVATE not write back (PASS, 0.080201s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.9199s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.9206s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.26723s) mmap test6: check mmap write/read content on two nodes (PASS, 0.31323s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.9962s) mmap test11: multi-thread shared mmap (PASS, 1.7478s) mmap test10: mtime not change for readonly mmap access (PASS, 1.152s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 11:59:16 (1772470756) oleg260-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 11:59:22 (1772470762) PASS 20 (10s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 11:59:32 (1772470772) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 11:59:41 (1772470781) now is 1772470784 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 new atime is 1772470845 PASS 23 (73s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 12:00:54 (1772470854) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 40.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 PASS 24a (10s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 12:01:04 (1772470864) PASS 24b (10s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 12:01:15 (1772470875) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (10s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 12:01:24 (1772470884) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (5s) == sanityn test 26a: allow mtime to get older ============ 12:01:29 (1772470889) utime: good mknod times 1772470890 <= 1772470891 <= 1772470891 for /mnt/lustre/f26a utime: good mknod times 1772470890 <= 1772470891 <= 1772470891 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 12:01:38 (1772470898) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 12:01:49 (1772470909) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-02 12:01:51.000000000 -0500 Birth: 2026-03-02 12:01:51.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-02 12:01:51.000000000 -0500 Birth: 2026-03-02 12:01:51.000000000 -0500 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 12:02:00 (1772470920) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 12:02:12 (1772470932) fail_loc=0x314 PASS 31a (11s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 12:02:22 (1772470942) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.160@tcp:/lustre 7542784 41984 7494656 1% /mnt/lustre2 PASS 31b (26s) == sanityn test 31r: open-rename(replace) race =========== 12:02:48 (1772470968) -rw-r--r-- 1 root root 0 Mar 2 12:02 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (13s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 12:03:01 (1772470981) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42994 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 12:03:11 (1772470991) PASS 31t (10s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 12:03:23 (1772471003) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 12:03:28 (1772471008) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 12:03:32 (1772471012) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 12:03:36 (1772471016) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (5s) == sanityn test 34: no lock timeout under IO ============= 12:03:41 (1772471021) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid in FULL state after 0 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid in IDLE state after 0 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid in FULL state after 0 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid in IDLE state after 0 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e8008cf6800.ost_server_uuid in FULL state after 0 sec oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e8008cf6800.ost_server_uuid in IDLE state after 0 sec PASS 34 (147s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 12:06:08 (1772471168) fail_loc=0x80000317 Race attempt 0 Wait for 49056 49068 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1746: 49056 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.160@tcp:/lustre 7.2G 42M 7.2G 1% /mnt/lustre 192.168.202.160@tcp:/lustre 7.2G 42M 7.2G 1% /mnt/lustre2 - open/close 481 (time 1772471186.95 total 10.03 last 47.96) - open/close 960 (time 1772471196.95 total 20.03 last 47.89) - open/close 1436 (time 1772471206.95 total 30.04 last 47.58) - open/close 1930 (time 1772471216.97 total 40.05 last 49.31) - open/close 2431 (time 1772471226.97 total 50.05 last 50.10) - open/close 2900 (time 1772471237.00 total 60.08 last 46.79) - open/close 3278 (time 1772471247.02 total 70.10 last 37.73) - open/close 3836 (time 1772471257.02 total 80.10 last 55.77) total: 4000 open/close in 82.40 seconds: 48.54 ops/second PASS 35 (91s) == sanityn test 36: handle ESTALE/open-unlink correctly == 12:07:39 (1772471259) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.916597 s, 57.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7495680) after_dd(7426048) after(7477248) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.793216 s, 66.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7477248) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.875032 s, 59.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7488512) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.996319 s, 52.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7479296) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.912585 s, 57.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7488512) after_dd(7422976) after(7474176) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.796052 s, 65.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7474176) after_dd(7419904) after(7471104) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.816474 s, 64.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7471104) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.808256 s, 64.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7489536) after_dd(7422976) after(7474176) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.823689 s, 63.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7474176) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00701 s, 52.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7479296) after_dd(7438336) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.67028 s, 78.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7425024) after(7495680) PASS 36 (316s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 12:12:55 (1772471575) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 - create 966 (time 1772471591.84 total 10.01 last 96.51) - create 1957 (time 1772471601.84 total 20.01 last 99.05) - create 3051 (time 1772471611.84 total 30.02 last 109.38) - create 4277 (time 1772471621.85 total 40.02 last 122.54) - create 5359 (time 1772471631.85 total 50.03 last 108.14) - create 6506 (time 1772471641.86 total 60.03 last 114.63) - create 7690 (time 1772471651.86 total 70.04 last 118.36) - create 8878 (time 1772471661.86 total 80.04 last 118.80) total: 10000 create in 88.49 seconds: 113.01 ops/second PASS 37 (104s) == sanityn test 39a: file mtime does not change after rename ========================================================== 12:14:39 (1772471679) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 12:14:47 (1772471687) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 12:14:56 (1772471696) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 12:15:07 (1772471707) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 12:15:17 (1772471717) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (22s) == sanityn test 40c: pdirops: link and others ======================================================================== 12:15:39 (1772471739) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (21s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 12:16:01 (1772471761) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 12:16:18 (1772471778) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 12:16:37 (1772471797) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 12:16:52 (1772471812) open: File exists Conflict PASS 41b (17s) == sanityn test 41c: pdirops: create vs link ======================================================================== 12:17:09 (1772471829) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (18s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 12:17:27 (1772471847) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 12:17:42 (1772471862) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 12:17:56 (1772471876) Conflict PASS 41f (17s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 12:18:14 (1772471894) Conflict PASS 41g (16s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 12:18:29 (1772471909) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 12:18:42 (1772471922) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1144s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 12:37:46 (1772473066) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 12:38:01 (1772473081) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 12:38:16 (1772473096) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 12:38:32 (1772473112) Conflict PASS 42d (16s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 12:38:48 (1772473128) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (18s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 12:39:06 (1772473146) Conflict PASS 42f (20s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 12:39:26 (1772473166) Conflict PASS 42g (19s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 12:39:45 (1772473185) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 12:40:00 (1772473200) PASS 43a (87s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 12:41:27 (1772473287) Conflict PASS 43b (16s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 12:41:43 (1772473303) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 12:41:59 (1772473319) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (19s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 12:42:18 (1772473338) Conflict PASS 43e (19s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 12:42:37 (1772473357) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (17s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 12:42:54 (1772473374) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 12:43:10 (1772473390) Conflict PASS 43h (17s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 12:43:27 (1772473407) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 12:43:30 (1772473410) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (129s) == sanityn test 43k: unlink vs create ==================== 12:45:39 (1772473539) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (851s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 12:59:50 (1772474390) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Mar 2 12:59:56 EST 2026 PASS 44a (9s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:00:00 (1772474400) open: File exists Conflict PASS 44b (8s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:00:08 (1772474408) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (9s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:00:17 (1772474417) Conflict PASS 44d (9s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:00:26 (1772474426) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:00:35 (1772474435) Conflict PASS 44f (10s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:00:45 (1772474445) Conflict PASS 44g (9s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:00:54 (1772474454) Conflict PASS 44h (10s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:01:04 (1772474464) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:01:06 (1772474466) PASS 45a (77s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:02:23 (1772474543) Conflict PASS 45b (9s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 13:02:32 (1772474552) Conflict PASS 45c (9s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 13:02:41 (1772474561) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (9s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 13:02:50 (1772474570) Conflict PASS 45e (9s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 13:02:59 (1772474579) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 13:03:09 (1772474589) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 13:03:19 (1772474599) Conflict PASS 45h (9s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 13:03:28 (1772474608) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 13:03:31 (1772474611) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (694s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 13:15:05 (1772475305) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 13:15:12 (1772475312) open: File exists Conflict PASS 46b (7s) == sanityn test 46c: pdirops: link vs link ======================================================================== 13:15:19 (1772475319) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 13:15:26 (1772475326) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 13:15:33 (1772475333) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 13:15:39 (1772475339) Conflict PASS 46f (8s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 13:15:47 (1772475347) Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 13:15:54 (1772475354) Conflict PASS 46h (7s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 13:16:01 (1772475361) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 13:16:02 (1772475362) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 13:16:04 (1772475364) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 13:16:05 (1772475365) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 13:16:07 (1772475367) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 13:16:08 (1772475368) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 13:16:09 (1772475369) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 13:16:11 (1772475371) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 13:16:12 (1772475372) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00702317 s, 1.5 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 13:16:20 (1772475380) 0+1 records in 0+1 records out 158 bytes copied, 0.0194909 s, 8.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216341) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 13:16:25 (1772475385) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00841211 s, 122 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396967 s, 26.4 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 13:16:41 (1772475401) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0177841 s, 57.6 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 13:16:49 (1772475409) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215241 s, 48.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 13:16:54 (1772475414) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 13:16:59 (1772475419) llite.lustre-ffff8e8008cf6800.xattr_cache=1 llite.lustre-ffff8e80202cc800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505008 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:00.000000000 -0500 Modify: 2026-03-02 13:17:00.000000000 -0500 Change: 2026-03-02 13:17:00.000000000 -0500 Birth: 2026-03-02 13:17:00.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:05.000000000 -0500 Modify: 2026-03-02 13:17:06.000000000 -0500 Change: 2026-03-02 13:17:06.000000000 -0500 Birth: 2026-03-02 13:17:05.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:11.000000000 -0500 Modify: 2026-03-02 13:17:11.000000000 -0500 Change: 2026-03-02 13:17:11.000000000 -0500 Birth: 2026-03-02 13:17:11.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:11.000000000 -0500 Modify: 2026-03-02 13:17:11.000000000 -0500 Change: 2026-03-02 13:17:11.000000000 -0500 Birth: 2026-03-02 13:17:11.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:16.000000000 -0500 Modify: 2026-03-02 13:17:17.000000000 -0500 Change: 2026-03-02 13:17:17.000000000 -0500 Birth: 2026-03-02 13:17:17.000000000 -0500 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:17:16.000000000 -0500 Modify: 2026-03-02 13:17:17.000000000 -0500 Change: 2026-03-02 13:17:17.000000000 -0500 Birth: 2026-03-02 13:17:17.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 13:17:24 (1772475444) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 13:17:33 (1772475453) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 13:17:41 (1772475461) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 13:17:55 (1772475475) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 13:18:06 (1772475486) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 13:18:07 (1772475487) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 13:18:21 (1772475501) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (16s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 13:18:37 (1772475517) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1772475518 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.2987 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (51s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 13:19:28 (1772475568) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1772475570 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.83894 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (49s) == sanityn test 60: Verify data_version behaviour ======== 13:20:17 (1772475617) PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 13:20:21 (1772475621) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 13:20:25 (1772475625) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 13:20:29 (1772475629) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00345014 s, 19.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 13:20:31 (1772475631) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0112217 s, 5.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 13:20:33 (1772475633) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 13:20:34 (1772475634) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00815129 s, 8.0 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 13:20:36 (1772475636) llite.lustre-ffff8e8008cf6800.xattr_cache=1 llite.lustre-ffff8e80202cc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 13:20:39 (1772475639) llite.lustre-ffff8e8008cf6800.xattr_cache=1 llite.lustre-ffff8e80202cc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 13:20:42 (1772475642) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00636159 s, 161 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0137902 s, 74.3 kB/s 229110: get lock2 229111: get lock1 229110: done 229110: get lock1 229111: done 229111: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229111: cannot get lock2: Resource deadlock avoided 229111: exit rc=0 229110: done 229110: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 13:20:49 (1772475649) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.017789 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115977 s, 90.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229692 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 13:20:57 (1772475657) opening files: ........ closing files: ........ PASS 76 (23s) == sanityn test 77a: check FIFO NRS policy =============== 13:21:20 (1772475680) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0845055 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287401 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328519 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128322 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359584 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143841 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261277 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440463 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272379 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244924 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661998 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673853 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597712 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787944 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558206 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762364 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822838 s, 12.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438766 s, 239 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261471 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025864 s, 40.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235356 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052938 s, 198 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155438 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340702 s, 30.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552484 s, 190 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287204 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013118 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243683 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244692 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262104 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111648 s, 93.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209747 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017512 s, 59.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 13:21:24 (1772475684) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118103 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138353 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375959 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351794 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357836 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376884 s, 27.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303344 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408584 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390181 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327864 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437778 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598539 s, 17.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704199 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813996 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523552 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697402 s, 15.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712644 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093451 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00825068 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01382 s, 75.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199793 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171632 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162596 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211997 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747101 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497754 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416297 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405973 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415539 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286904 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290279 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117106 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102787 s, 102 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0788104 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018289 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335971 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324591 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353831 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321193 s, 32.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450342 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433839 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570147 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554001 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432524 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535512 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560281 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488398 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467117 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475769 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046997 s, 22.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183635 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144648 s, 72.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173226 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017784 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174907 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220664 s, 47.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215151 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205673 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236131 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114924 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031026 s, 33.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018658 s, 56.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023506 s, 44.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202763 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146374 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140513 s, 74.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 13:21:28 (1772475688) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110007 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247264 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310805 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516983 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050796 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054068 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708514 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481815 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400461 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687871 s, 15.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978943 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798987 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976362 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846405 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115016 s, 9.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110325 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133348 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250899 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242357 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348388 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267543 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261498 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271367 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231983 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193204 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284978 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198628 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268815 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251315 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319464 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030835 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050508 s, 208 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356589 s, 29.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113844 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117964 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118652 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221776 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204987 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265944 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108185 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262329 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299346 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483644 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375659 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577769 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504289 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427462 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423434 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414352 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056568 s, 18.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169185 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972879 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131872 s, 79.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116719 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258895 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601102 s, 174 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391136 s, 268 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302103 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397478 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105024 s, 99.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872612 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157786 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271121 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258176 s, 406 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00893384 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166532 s, 63.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0996975 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016998 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328502 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418339 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449119 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057519 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229401 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455014 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519428 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613133 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079062 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673252 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064027 s, 16.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631694 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833908 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678028 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853752 s, 12.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507499 s, 207 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685973 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219255 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201952 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133266 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144921 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012758 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247178 s, 424 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377729 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496427 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037546 s, 279 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447557 s, 234 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395009 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125194 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402502 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398582 s, 263 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 13:21:34 (1772475694) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1562 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112877 s, 92.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388384 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368334 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340445 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353487 s, 29.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454315 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048096 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464958 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427285 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270991 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594715 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422221 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543176 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591289 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532079 s, 19.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675687 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248285 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155834 s, 67.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158602 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272016 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287281 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200704 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320824 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122025 s, 85.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027988 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246313 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273164 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105711 s, 99.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036207 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00592565 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145195 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741875 s, 141 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.10058 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123228 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181392 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330633 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034181 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307098 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453947 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400269 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376016 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304207 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456489 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282317 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534474 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018814 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423107 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341928 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333032 s, 31.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907292 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096338 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245578 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408024 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192627 s, 54.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219089 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273623 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486579 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311935 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220892 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160307 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115889 s, 90.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986734 s, 106 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104839 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571351 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303083 s, 346 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0881073 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183371 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304892 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535422 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403551 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030322 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479956 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054965 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637683 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616884 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062521 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834488 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581465 s, 18.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074207 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552347 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875544 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842718 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150224 s, 69.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342513 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360802 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249029 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025552 s, 41.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325927 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335386 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036079 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373673 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361774 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286933 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297471 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321445 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034591 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313834 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349271 s, 30.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 13:21:40 (1772475700) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.60@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0770942 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176005 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190354 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119582 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246814 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435045 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380135 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481132 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037162 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393056 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354113 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478886 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402494 s, 26.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319955 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350652 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318822 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575624 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138327 s, 75.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130985 s, 80.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126262 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187589 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153566 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238383 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202406 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136218 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167869 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243164 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057246 s, 183 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459932 s, 228 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400153 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248335 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256956 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395113 s, 265 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0645411 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123069 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357709 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350433 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335447 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033398 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248371 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352235 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195845 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294871 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463448 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537575 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501759 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437668 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468859 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427663 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494151 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01453 s, 72.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243189 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215332 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202618 s, 51.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184926 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236292 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199575 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179776 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159928 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274025 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219564 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233879 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00933915 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254746 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408215 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245366 s, 427 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110013 s, 153 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161698 s, 64.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213976 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379338 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103152 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455565 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426448 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301246 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443861 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360475 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581347 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545534 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457441 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407268 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457338 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560417 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423169 s, 24.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179236 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195327 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210591 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172037 s, 61.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144328 s, 72.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199688 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278335 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125426 s, 83.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038119 s, 275 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098147 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016004 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986997 s, 106 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195044 s, 53.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248852 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210989 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158228 s, 66.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0842359 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023746 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348978 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564006 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765694 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819318 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734517 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761901 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722798 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712048 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756564 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911701 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716234 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754537 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882602 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085439 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872486 s, 12.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296931 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238804 s, 43.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262087 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147859 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254199 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014675 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254289 s, 41.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243659 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136987 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202601 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110621 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174705 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431699 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252474 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161676 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00848354 s, 124 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 13:21:49 (1772475709) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.219423 s, 76.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176021 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254658 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253585 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365026 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355425 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506271 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328153 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063951 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721822 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681991 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783725 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769575 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707119 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552936 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489054 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782699 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268639 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018271 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277501 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230148 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236271 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250577 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247486 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193131 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241551 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265299 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299222 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030377 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331681 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353956 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326685 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405346 s, 25.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0997809 s, 168 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341955 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391347 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023741 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312945 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234603 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213104 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292613 s, 35.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234016 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196512 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861106 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819004 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880132 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788668 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772584 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948068 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910816 s, 11.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00992328 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123724 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00655217 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416379 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225709 s, 46.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421244 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544772 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591353 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380425 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277033 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425761 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195744 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016475 s, 63.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575504 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175298 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419743 s, 25.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.091842 s, 183 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137217 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034804 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426018 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257568 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314581 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477283 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495801 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046724 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506966 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540227 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512376 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032824 s, 31.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380538 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479659 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443889 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600777 s, 17.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958935 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236012 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243213 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371268 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00315062 s, 333 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479035 s, 21.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506868 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713216 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366744 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475467 s, 22.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602043 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230486 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572012 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150844 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165263 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126943 s, 82.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117412 s, 143 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244542 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230859 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411219 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554528 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459037 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552105 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476995 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397386 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700871 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580922 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670364 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697589 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768932 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705366 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725608 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636744 s, 16.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80202cc800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e80202cc800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217826 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028221 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034736 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156539 s, 67.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215701 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224438 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238709 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258149 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029961 s, 35.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269182 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362884 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345023 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380265 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339479 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342437 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382316 s, 27.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 13:21:59 (1772475719) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 13:22:04 (1772475724) oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg260-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg260-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg260-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg260-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 13:22:09 (1772475729) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (11s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 13:22:20 (1772475740) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301445 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206106 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234828 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181282 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571531 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389573 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567804 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709638 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473263 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646217 s, 16.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599159 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943179 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10377 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217695 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208085 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305151 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309474 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269256 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358842 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257309 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441279 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275901 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326027 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272936 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477303 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038955 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287413 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449641 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395106 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367311 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368263 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363869 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384721 s, 27.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00968 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4821 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (48s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 13:23:08 (1772475788) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24494 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175282 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230559 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039261 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209771 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301842 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035651 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307451 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151775 s, 69.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127917 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127956 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117098 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119969 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145779 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118772 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154577 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119183 s, 8.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279673 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261385 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025579 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320536 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277872 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342885 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03464 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270332 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03746 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331824 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337782 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354842 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415045 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373333 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368291 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481606 s, 21.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7558 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9543 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22643 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307536 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321928 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031126 s, 33.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027946 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264021 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210833 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309202 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192159 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150373 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150427 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150248 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172193 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14922 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152593 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182019 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184294 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289052 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283652 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263272 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334271 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323308 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356419 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300913 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365637 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404916 s, 25.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035727 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415926 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321265 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369992 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357895 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377266 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429694 s, 24.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7114 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9607 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (136s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 13:25:24 (1772475924) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21697 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02483 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261322 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286028 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276019 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562292 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431344 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490857 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413642 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161174 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171383 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166395 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187772 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182753 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165587 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170913 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238352 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256919 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216483 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208978 s, 50.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231458 s, 45.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304447 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330654 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227594 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245772 s, 42.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382182 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390712 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357688 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364837 s, 28.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345194 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337855 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374629 s, 28.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8397 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9476 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21822 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251529 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317971 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217248 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02538 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245057 s, 42.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328436 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126683 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267253 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159257 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159314 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158144 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161032 s, 6.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176141 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15005 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193628 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182567 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220333 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241249 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266996 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269008 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278849 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198042 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021403 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021592 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276073 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031741 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303994 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288748 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363186 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354431 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565327 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617547 s, 170 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8342 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.949 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (135s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 13:27:40 (1772476060) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251945 s, 66.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173381 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207867 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111113 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141513 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028579 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202281 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917459 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507208 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050762 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551729 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385242 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453317 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658915 s, 15.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604943 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621441 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737563 s, 14.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129026 s, 81.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155478 s, 67.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019343 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179383 s, 58.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211366 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200856 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261547 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190977 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234066 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208313 s, 50.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307708 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216704 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244171 s, 42.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237444 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241753 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236415 s, 44.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95097 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73976 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.60@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.60@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.223875 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197627 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257428 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315996 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214633 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375306 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218181 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380441 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285981 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224718 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303445 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310008 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737781 s, 142 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258642 s, 4.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258359 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298682 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311968 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159087 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153823 s, 68.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192026 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227809 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015848 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023566 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287323 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377356 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158149 s, 66.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208783 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379806 s, 276 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199767 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236228 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382435 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408242 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286163 s, 36.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91913 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73977 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.60@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.232217 s, 72.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195779 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038042 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040034 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363936 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268845 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328243 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473996 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508314 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181897 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617607 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594898 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748604 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556967 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732316 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831692 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835136 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176511 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189582 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174586 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020322 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226639 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163268 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205976 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253558 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185667 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412246 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190274 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224324 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217136 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215971 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038364 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563684 s, 186 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93912 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88939 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.460755 s, 36.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245785 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293538 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353482 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282787 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460813 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449997 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594559 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606652 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410856 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771523 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838661 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880473 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305077 s, 3.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303697 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34733 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354573 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158532 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195679 s, 53.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195957 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192153 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164186 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196864 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247069 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207945 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247623 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190483 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251597 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246106 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258782 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268273 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418385 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288862 s, 36.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84376 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.744 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92632 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74151 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.830898 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254283 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299722 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251862 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250825 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200098 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270492 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302013 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389264 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153926 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173828 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160997 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156379 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186877 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164275 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159237 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18742 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159811 s, 65.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171978 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126659 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214614 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375709 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254033 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237353 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223231 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252348 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286318 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050802 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365487 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309289 s, 33.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305711 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523282 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466273 s, 22.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6525 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4354 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227 s, 73.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184352 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200272 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218777 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337869 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411914 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334441 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444805 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376821 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439751 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381102 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047027 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339028 s, 30.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640031 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520148 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499457 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614163 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100575 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125128 s, 83.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143171 s, 73.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172868 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170709 s, 61.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201618 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00743565 s, 141 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179699 s, 58.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143434 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035576 s, 295 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023398 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874861 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024299 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188649 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548183 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252129 s, 416 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93473 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75244 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428276 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313705 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247521 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446061 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220945 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272356 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292551 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380284 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577793 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580066 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025305 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205376 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199706 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236723 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230216 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457768 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463056 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016296 s, 64.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180681 s, 58.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017895 s, 58.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368039 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188816 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210205 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215061 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236315 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319079 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227328 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275311 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444585 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027861 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429243 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274781 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294783 s, 35.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77274 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74914 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77086 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73603 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (354s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 13:33:33 (1772476413) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254992 s, 65.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153401 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015351 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291543 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198001 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157843 s, 66.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154562 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270139 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287159 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491593 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286771 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470868 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345088 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045691 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05809 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266073 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629698 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140232 s, 74.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183315 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198179 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204112 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026655 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212973 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173848 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295369 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333376 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206833 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266175 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265656 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028296 s, 37.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292402 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247003 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291593 s, 36.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95075 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7497 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 13:34:10 (1772476450) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.60@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.60@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.229257 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169295 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181842 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207645 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340234 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273812 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358653 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283774 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162628 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208984 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317084 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288308 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159715 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242715 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254117 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299581 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314422 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150335 s, 69.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220219 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01777 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140875 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162486 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115825 s, 90.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244969 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188174 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023247 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231543 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279358 s, 375 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829795 s, 126 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755745 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121259 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842596 s, 124 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94395 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7419 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (39s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 13:34:49 (1772476489) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.60@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249792 s, 67.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134518 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299809 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305717 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326081 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346909 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360342 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382978 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326416 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144961 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288797 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436679 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381108 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04958 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040734 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478172 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410062 s, 25.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011716 s, 89.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112456 s, 93.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254929 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216807 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220316 s, 47.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207514 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314535 s, 33.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213591 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296165 s, 35.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193785 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243089 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023693 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269274 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266702 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233178 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303259 s, 34.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95139 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97593 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (34s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 13:35:23 (1772476523) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428586 s, 39.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181222 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146007 s, 71.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237889 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187996 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028548 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337074 s, 31.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289772 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293724 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664534 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086272 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099202 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942671 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223911 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224262 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269578 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266844 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889538 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162127 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129667 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166802 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119738 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213301 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020481 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161398 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235892 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025475 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278291 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311557 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244623 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232812 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216612 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242666 s, 43.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83612 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74181 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9349 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74029 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (76s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 13:36:39 (1772476599) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.830316 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201958 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298699 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326166 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053311 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055397 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438896 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427636 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591498 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164088 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172348 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158106 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160677 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18856 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174277 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183369 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196177 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135612 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015883 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996623 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407288 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608447 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471774 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040583 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319301 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372713 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427116 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011838 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136629 s, 76.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916555 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386273 s, 271 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120584 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647301 s, 162 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6735 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9521 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (64s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 13:37:43 (1772476663) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.229279 s, 73.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975329 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158565 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201294 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343086 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351767 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039673 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426285 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318365 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259246 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309868 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436169 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049242 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031336 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247141 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476522 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215399 s, 48.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014906 s, 70.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177262 s, 59.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103394 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181262 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110007 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246296 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243523 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249072 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235303 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216824 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025755 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738084 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200419 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235713 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021232 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441438 s, 238 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94023 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73088 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg260-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.420935 s, 39.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199973 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032896 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281242 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238978 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370128 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300108 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231237 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290229 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659703 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711471 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814241 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607311 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307625 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323548 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364692 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362582 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360045 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818519 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127271 s, 82.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124945 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207814 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162946 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167427 s, 62.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155253 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192259 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00854344 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00908788 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685119 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021544 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222756 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568863 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399674 s, 262 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg260-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79202 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73056 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77818 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73686 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (114s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 13:39:38 (1772476778) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre opts: Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg260-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.60,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg260-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.60,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg260-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.60,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/60766/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.60,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg260-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e8008917800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8e80091b3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e8008917800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e80091b3000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255667 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221074 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222916 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03201 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536102 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581573 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055899 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810211 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814419 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073551 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093092 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873493 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930032 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232484 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240457 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28857 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30664 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106731 s, 98.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222772 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200065 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019233 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183642 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231482 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174441 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021777 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178056 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248821 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219949 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235133 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02487 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486828 s, 215 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291024 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259335 s, 40.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95672 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97618 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249344 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231177 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196632 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620975 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678553 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845421 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783145 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065694 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664938 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925191 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790123 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904142 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100078 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26356 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25916 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293438 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291273 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142895 s, 73.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197239 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145289 s, 72.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870496 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165113 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220084 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968097 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191737 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392707 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204465 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019325 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198491 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381607 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219497 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302791 s, 346 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034543 s, 304 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93424 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74159 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (134s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 13:41:51 (1772476911) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg260-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:41:59 (1772476919) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:42:03 (1772476923) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg260-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.74158 s, 280 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg260-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0114 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:42:54 (1772476974) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132159 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234629 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480952 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419101 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336225 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623432 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711337 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685837 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833716 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796966 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748259 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948685 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977035 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081321 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824198 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964937 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101642 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e80091b3000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e80091b3000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199229 s, 52.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207056 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123587 s, 84.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234358 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128921 s, 81.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221648 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218334 s, 48.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208919 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222036 s, 47.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00610887 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178524 s, 58.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013551 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143296 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785203 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00678942 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019778 s, 53.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e80091b3000.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94681 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97161 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.184209 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255498 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293999 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126752 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11278 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132057 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114312 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131456 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129066 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14414 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141231 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14636 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147102 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138863 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134802 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184393 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194656 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137444 s, 76.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139143 s, 75.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194849 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169345 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188867 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196746 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157341 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125412 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228258 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193194 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176253 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171457 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250168 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209574 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219926 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268539 s, 39.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg260-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg260-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9642 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97471 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (68s) == sanityn test 77o: Changing rank should not panic ====== 13:44:02 (1772477042) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:44:07 (1772477047) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (40s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:44:46 (1772477086) mds.MDS.mdt.nrs_policies=tbf oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg260-client: oleg260-server: ssh exited with exit code 36 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 oleg260-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg260-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 13:44:58 (1772477098) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269743 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:45:40 (1772477140) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 13:45:43 (1772477143) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935330 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:45:44.000000000 -0500 Modify: 2026-03-02 13:45:44.000000000 -0500 Change: 2026-03-02 13:45:44.000000000 -0500 Birth: 2026-03-02 13:45:44.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:45:56 (1772477156) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:45:57 (1772477157) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:45:58 (1772477158) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:45:59 (1772477159) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:46:00 (1772477160) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 13:46:01 (1772477161) total: 2500 open/close in 5.40 seconds: 463.35 ops/second snapshot_time 1772477187.100641889 secs.nsecs start_time 1772477163.010842141 secs.nsecs elapsed_time 24.089799748 secs.nsecs open 2505 samples [usecs] 120 1700 815699 280443451 close 2504 samples [usecs] 64 1100 257624 29003408 mknod 2501 samples [usecs] 168 1594 710859 214650061 unlink 2500 samples [usecs] 264 16688 1265643 1117760447 mkdir 2 samples [usecs] 820 825 1645 1353025 rmdir 2 samples [usecs] 2085 3616 5701 17422681 rename 2500 samples [usecs] 266 17470 3474271 7529152375 getattr 3760 samples [usecs] 6 213 91592 2694040 parallel_rename_file 2500 samples [usecs] 266 17470 3474271 7529152375 crossdir_rename 2500 samples [usecs] 266 17470 3474271 7529152375 PASS 81d (27s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:46:28 (1772477188) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7583 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7583 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:46:31 (1772477191) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:46:32 (1772477192) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 13:46:39 (1772477199) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1772477200 Finishing test test1 at 1772477201 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:46:42 (1772477202) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 13:46:43 (1772477203) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:46:44 (1772477204) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:46:45 (1772477205) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5997 0x176d 0x240000400 1 5961 0x1749 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5998 0x176e 0x240000400 1 5962 0x174a 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 13:46:54 (1772477214) osc.lustre-OST0000-osc-ffff8e8008917800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8e80091b3000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8e8008917800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8e80091b3000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0460464 s, 178 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09283 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 288515 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148589 s, 276 kB/s osc.lustre-OST0000-osc-ffff8e8008917800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8e80091b3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e8008917800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e80091b3000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 13:47:05 (1772477225) llite.lustre-ffff8e8008917800.fast_read=0 llite.lustre-ffff8e80091b3000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0181787 s, 115 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8e8008917800.fast_read=1 llite.lustre-ffff8e80091b3000.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:47:19 (1772477239) llite.lustre-ffff8e8008917800.fast_read=0 llite.lustre-ffff8e80091b3000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00155181 s, 7.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8e8008917800.fast_read=1 llite.lustre-ffff8e80091b3000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:47:27 (1772477247) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:47:28 (1772477248) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00192768 s, 2.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:47:30 (1772477250) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.042831 s, 49.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:47:33 (1772477253) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0499685 s, 42.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 13:47:35 (1772477255) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00231564 s, 5.2 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 13:47:37 (1772477257) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00128126 s, 3.2 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 13:47:39 (1772477259) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00155261 s, 2.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 13:47:41 (1772477261) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00155398 s, 2.6 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 13:47:45 (1772477265) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1772477266 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1772477268 Finishing test test23 at 1772477268 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:47:52 (1772477272) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 13:47:53 (1772477273) fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 13:48:11 (1772477291) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:48:12 (1772477292) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00462011 s, 222 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937865 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:48:12.000000000 -0500 Modify: 2026-03-02 13:48:12.000000000 -0500 Change: 2026-03-02 13:48:12.000000000 -0500 Birth: 2026-03-02 13:48:12.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937865 -rw-r--r-- 1 0 0 1772477292 1772477292 1772477292 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937865 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:48:12.000000000 -0500 Modify: 2026-03-02 13:48:12.000000000 -0500 Change: 2026-03-02 13:48:12.000000000 -0500 Birth: 2026-03-02 13:48:12.000000000 -0500 1772477292 1024 11 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 13:48:14 (1772477294) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 13:48:16 (1772477296) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0962393 s, 109 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5968 0x1750 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:48:21 (1772477301) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0931858 s, 113 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6007 0x1777 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7583 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 13:48:29 (1772477309) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00163237 s, 6.1 MB/s llite.lustre-ffff8e8008917800.hybrid_io=0 llite.lustre-ffff8e80091b3000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.24488 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:48:36 (1772477316) 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 192.168.202.160@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 PASS 109 (489s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:56:45 (1772477805) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 13:56:45.000000000 -0500 Modify: 2026-03-02 13:56:45.000000000 -0500 Change: 2026-03-02 13:56:45.000000000 -0500 Birth: 2026-03-02 13:56:45.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg260-client.virtnet oleg260-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:57:50 (1772477870) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 13:57:51 (1772477871) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 13:57:53 (1772477873) lustre-OST0000-osc-ffff8e800a02d000 active. lustre-OST0001-osc-ffff8e800a02d000 active. lustre-OST0000-osc-ffff8e800a02d000 active. lustre-OST0001-osc-ffff8e800a02d000 active. MGC192.168.202.160@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 13:57:55 (1772477875) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 13:57:56 (1772477876) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 13:57:57 (1772477877) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 13:57:58 (1772477878) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00621313 s, 3.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162461 s, 1.3 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 13:58:03 (1772477883) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0842 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0879 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0979 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.085 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0909 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.092 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0899 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1377 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1603 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1451 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1525 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1723 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1644 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1623 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2254 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2431 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.215 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2267 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e8008eff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e8008eff800.ost_server_uuid in IDLE state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 8118 sec ============== 13:59:10 (1772477950) === sanityn: start cleanup 13:59:10 (1772477950) === Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg260-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 13:59:54 (1772477994) ===