-----============= acceptance-small: sanityn ============----- Sun Nov 16 22:25:42 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg254-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 22:26:02 (1763349962) === Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Mount client oleg254-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg254-client.virtnet: executing check_config_client /mnt/lustre oleg254-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg254-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cd2050d8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cd209739000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd2050d8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd209739000.idle_timeout=debug disable quota as required oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 22:26:31 (1763349991) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422395 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7582] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 22:26:34 (1763349994) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 22:26:41 (1763350001) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 22:26:48 (1763350008) -rw-r--r-- 1 root root 0 Nov 16 22:26 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 22:26:56 (1763350016) -rw-r--r-- 1 root root 0 Nov 16 22:26 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 22:27:05 (1763350025) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 22:27:13 (1763350033) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 22:27:21 (1763350041) total 1 -rwxrwxrwx 1 root root 0 Nov 16 22:26 f2a -rwxrwxrwx 1 root root 0 Nov 16 22:26 f2b total 1 -rwxrwxrwx 1 root root 0 Nov 16 22:26 f2a -rwxrwxrwx 1 root root 0 Nov 16 22:26 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0220485 s, 23.2 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 22:27:29 (1763350049) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 22:27:33 (1763350053) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.490247 s, 4.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 22:27:40 (1763350060) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 22:27:46 (1763350066) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 22:27:54 (1763350074) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 22:28:01 (1763350081) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 22:28:08 (1763350088) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 22:28:13 (1763350093) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 22:28:21 (1763350101) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 22:28:28 (1763350108) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666496 s, 15.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0360311 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 22:28:34 (1763350114) 1+0 records in 1+0 records out 1 byte copied, 0.012927 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0262096 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0187199 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0342762 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104127 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0162123 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0165055 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132757 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113926 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111914 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0137662 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.022096 s, 0.0 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 22:28:42 (1763350122) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0259818 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0533703 s, 76.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00024133 s, 12.7 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 22:28:49 (1763350129) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 22:28:55 (1763350135) warning: '-runtime' deprecated, use '-t runtime' instead - link 410 (time 1763350147.52 total 10.04 last 40.85) - link 791 (time 1763350157.52 total 20.04 last 38.10) - link 1171 (time 1763350167.54 total 30.06 last 37.91) - link 1542 (time 1763350177.57 total 40.09 last 36.98) - link 1915 (time 1763350187.58 total 50.10 last 37.26) - link 2310 (time 1763350197.60 total 60.12 last 39.42) - link 2696 (time 1763350207.62 total 70.14 last 38.55) - link 3049 (time 1763350217.64 total 80.16 last 35.22) - link 3410 (time 1763350227.64 total 90.16 last 36.09) total: 3752 link in 100.00 seconds: 37.52 ops/second using seed 860904200 running for 100 seconds - stat 8298 (time 1763350149 ; total 11 ; last 11) - stat 10000 (time 1763350151 ; total 13 ; last 2) - stat 19516 (time 1763350162 ; total 24 ; last 11) - stat 20000 (time 1763350162 ; total 24 ; last 0) - stat 28056 (time 1763350173 ; total 35 ; last 11) - stat 30000 (time 1763350175 ; total 37 ; last 2) - stat 38806 (time 1763350186 ; total 48 ; last 11) - stat 40000 (time 1763350187 ; total 49 ; last 1) - stat 44646 (time 1763350198 ; total 60 ; last 11) - stat 49180 (time 1763350209 ; total 71 ; last 11) - stat 50000 (time 1763350211 ; total 73 ; last 2) - stat 53907 (time 1763350222 ; total 84 ; last 11) - stat 58495 (time 1763350233 ; total 95 ; last 11) - stat 60000 (time 1763350235 ; total 97 ; last 2) total: 60779 stats in 100 seconds: 607.789978 stats/second - unlinked 0 (time 1763350188 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3752) error: No such file or directory total: 3751 unlinks in 92 seconds: 40.771740 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21374) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21376) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 22:31:26 (1763350286) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 22:31:34 (1763350294) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 22562 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 22:31:41 (1763350301) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 23142 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 22:31:48 (1763350308) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23720 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 22:31:54 (1763350314) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24302 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 22:32:01 (1763350321) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24884 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 22:32:08 (1763350328) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (48s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 22:32:57 (1763350377) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.552919 s, 15.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.388147 s, 21.6 MB/s Chance of close/open is 1 in 50 Seed set to 381 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x10b0aa 000006[0] 1763350382.349732 trunc from 00000000 to 0x10b0a9 (0x10b0aa bytes) 000007[0] 1763350382.453631 mapwrite 0x3bc244 thru 0x3c7207 (0xafc4 bytes) 000009[0] 1763350382.600494 read 0x22ef97 thru 0x238a94 (0x9afe bytes) truncating to largest ever: 0x53ff38 000010[1] 1763350382.644588 trunc from 0x3c7208 to 0x53ff37 (0x178d30 bytes) truncating to largest ever: 0x900756 000011[1] 1763350382.701454 trunc from 0x53ff38 to 0x900755 (0x3c081e bytes) 000012[0] 1763350382.723349 read 0x89ae67 thru 0x8a1949 (0x6ae3 bytes) 000013[1] 1763350382.782345 read 0x0ddaad thru 0x0e01b9 (0x270d bytes) 000014[0] 1763350382.793158 mapread 0x4aebda thru 0x4af7da (0xc01 bytes) 000015[0] 1763350382.830117 read 0x6e2457 thru 0x6e285b (0x405 bytes) 000017[0] 1763350382.838448 mapwrite 0x84bdd6 thru 0x853863 (0x7a8e bytes) 000018[1] 1763350382.924106 trunc from 0x900756 to 0x3b2665 (0x54e0f0 bytes) 000019[0] 1763350382.977933 mapread 0x35ef1d thru 0x364ed3 (0x5fb7 bytes) 000021[0] 1763350383.020920 trunc from 0x3b2666 to 0x191af4 (0x220b71 bytes) 000023[1] 1763350383.109954 read 0x18382c thru 0x191af4 (0xe2c9 bytes) 000024[1] 1763350383.152391 write 0x2efc6c thru 0x2fad1f (0xb0b4 bytes) 000030[0] 1763350383.273457 mapwrite 0x58093c thru 0x589872 (0x8f37 bytes) 000032[1] 1763350383.393039 read 0x057f8f thru 0x067e90 (0xff02 bytes) 000033[0] 1763350383.421565 write 0x03cc9e thru 0x04400e (0x7371 bytes) 000035[1] 1763350383.492195 mapwrite 0x7dbef5 thru 0x7dfe29 (0x3f35 bytes) 000036[1] 1763350383.596013 mapread 0x762d67 thru 0x76675a (0x39f4 bytes) 000037[1] 1763350383.614584 trunc from 0x7dfe2a to 0x79fd06 (0x40123 bytes) 000039[1] 1763350383.671735 write 0x85ae12 thru 0x865d05 (0xaef4 bytes) 000040[1] 1763350383.681437 mapread 0x7a4db3 thru 0x7a91d9 (0x4427 bytes) 000043[0] 1763350383.729598 mapread 0x0ef5f0 thru 0x0fd981 (0xe392 bytes) 000044[1] 1763350383.794058 read 0x537161 thru 0x545fb3 (0xee53 bytes) 000045[1] 1763350383.822399 trunc from 0x865d06 to 0x2810ed (0x5e4c18 bytes) 000046[0] 1763350383.863914 mapwrite 0x438a8f thru 0x439a4a (0xfbc bytes) 000047[0] 1763350383.947665 read 0x1f49c0 thru 0x20152b (0xcb6c bytes) 000052[0] 1763350384.045267 mapwrite 0x0b478a thru 0x0c05d6 (0xbe4d bytes) 000053[0] 1763350384.130038 read 0x24df98 thru 0x24dfa5 (0xe bytes) 000054[1] 1763350384.155644 read 0x3e7cfc thru 0x3f6bfe (0xef03 bytes) 000055[0] 1763350384.189405 mapread 0x0b7a48 thru 0x0c05a0 (0x8b59 bytes) 000057[0] 1763350384.215869 trunc from 0x439a4b to 0x7705de (0x336b94 bytes) 000058[0] 1763350384.301336 mapwrite 0x99cc62 thru 0x9a9f99 (0xd338 bytes) 000059[1] 1763350384.419946 mapread 0x0f89e4 thru 0x0fa802 (0x1e1f bytes) 000064[1] 1763350384.577216 mapread 0x332822 thru 0x332be2 (0x3c1 bytes) 000066[0] 1763350384.617074 trunc from 0x9a9f9a to 0x80c012 (0x19df87 bytes) 000067[0] 1763350384.658360 trunc from 0x80c013 to 0x164c42 (0x6a73d0 bytes) 000069[1] 1763350384.714212 trunc from 0x164c43 to 0x162737 (0x250b bytes) 000072[0] 1763350384.814107 write 0x4c4832 thru 0x4d1239 (0xca08 bytes) 000073[0] 1763350384.860675 mapwrite 0x5eb54c thru 0x5f8f80 (0xda35 bytes) 000075[1] 1763350385.028219 write 0x5a57d1 thru 0x5a9eb3 (0x46e3 bytes) 000079[0] 1763350385.110236 write 0x94aabb thru 0x954dbc (0xa302 bytes) 000080[0] 1763350385.127153 trunc from 0x954dbd to 0x849595 (0x10b827 bytes) 000081[1] 1763350385.167106 mapread 0x62705a thru 0x62b707 (0x46ae bytes) 000082[0] 1763350385.212565 write 0x028acf thru 0x02a027 (0x1559 bytes) 000083[0] 1763350385.259647 read 0x535eff thru 0x543119 (0xd21b bytes) 000084[1] 1763350385.278889 write 0x07e43e thru 0x088741 (0xa304 bytes) 000086[1] 1763350385.345968 read 0x58d905 thru 0x5930ff (0x57fb bytes) 000087[1] 1763350385.365044 mapwrite 0x22d15e thru 0x22e139 (0xfdc bytes) 000088[0] 1763350385.413931 trunc from 0x849596 to 0x50e482 (0x33b113 bytes) 000089[1] 1763350385.479504 trunc from 0x50e483 to 0x2a8696 (0x265dec bytes) 000090[1] 1763350385.510220 mapwrite 0x5f33c4 thru 0x5fbc5b (0x8898 bytes) 000091[1] 1763350385.612142 trunc from 0x5fbc5c to 0x7b34fc (0x1b78a1 bytes) 000096[0] 1763350385.715133 trunc from 0x7b34fd to 0x1a8dc6 (0x60a736 bytes) 000098[1] 1763350385.779415 mapread 0x01e52e thru 0x01ff68 (0x1a3b bytes) 000103[0] 1763350385.895366 trunc from 0x1a8dc7 to 0x3bef00 (0x21613a bytes) 000104[1] 1763350385.937193 read 0x2eb382 thru 0x2f3aca (0x8749 bytes) 000108[0] 1763350386.109029 mapwrite 0x29d178 thru 0x2a9c69 (0xcaf2 bytes) 000109[0] 1763350386.209227 mapread 0x3b5a4b thru 0x3b9221 (0x37d7 bytes) 000110[0] 1763350386.241964 write 0x9cbb06 thru 0x9cc8dc (0xdd7 bytes) 000111[0] 1763350386.261984 read 0x29145e thru 0x297b11 (0x66b4 bytes) 000112[0] 1763350386.270605 trunc from 0x9cc8dd to 0x631388 (0x39b554 bytes) 000113[1] 1763350386.286202 trunc from 0x631389 to 0x46cb2a (0x1c485e bytes) 000114[0] 1763350386.328310 write 0x80cd10 thru 0x8118c4 (0x4bb5 bytes) 000115[1] 1763350386.353094 mapwrite 0x9780b1 thru 0x984d98 (0xcce8 bytes) 000121[0] 1763350386.699501 write 0x4c66a9 thru 0x4c7093 (0x9eb bytes) 000123[0] 1763350386.813578 read 0x81f82d thru 0x82d6d6 (0xdeaa bytes) 000124[0] 1763350386.857017 trunc from 0x984d99 to 0x350c55 (0x634143 bytes) 000125[0] 1763350386.906044 write 0x46027d thru 0x46cd9e (0xcb22 bytes) 000126[0] 1763350386.911615 mapread 0x1a344b thru 0x1ada58 (0xa60e bytes) 000127[1] 1763350386.927587 write 0x9c9ed5 thru 0x9d09e0 (0x6b0c bytes) 000128[0] 1763350386.959456 mapwrite 0x0998e5 thru 0x0a1e91 (0x85ad bytes) 000129[1] 1763350387.062324 mapread 0x149ac7 thru 0x14b3d5 (0x190f bytes) 000130[1] 1763350387.120421 mapwrite 0x967e05 thru 0x96f7f9 (0x79f5 bytes) 000133[0] 1763350387.306778 trunc from 0x9d09e1 to 0x5e1b57 (0x3eee89 bytes) 000135[0] 1763350387.345012 write 0x5d082d thru 0x5d2514 (0x1ce8 bytes) 000137[0] 1763350387.355086 write 0x11ecd6 thru 0x126bcf (0x7efa bytes) 000140[0] 1763350387.431119 mapread 0x1e36ca thru 0x1eccab (0x95e2 bytes) 000141[1] 1763350387.480724 mapwrite 0x39c266 thru 0x3a1190 (0x4f2b bytes) 000143[0] 1763350387.662491 mapread 0x18a547 thru 0x18d23b (0x2cf5 bytes) 000149[0] 1763350387.743445 mapwrite 0x508559 thru 0x50afb5 (0x2a5d bytes) 000150[1] 1763350387.868238 mapwrite 0x58bf1a thru 0x58c6a7 (0x78e bytes) 000151[1] 1763350387.960179 write 0x1597aa thru 0x15ed03 (0x555a bytes) 000155[0] 1763350388.029272 mapwrite 0x1170dc thru 0x117604 (0x529 bytes) 000157[0] 1763350388.149238 read 0x59ff92 thru 0x5a2be7 (0x2c56 bytes) 000160[1] 1763350388.197961 trunc from 0x5e1b58 to 0x14b751 (0x496406 bytes) 000161[0] 1763350388.295935 write 0x22f8a9 thru 0x231a2c (0x2184 bytes) 000162[1] 1763350388.384977 mapread 0x1d50bc thru 0x1d8866 (0x37ab bytes) 000163[0] 1763350388.431149 write 0x966a07 thru 0x9674cc (0xac6 bytes) 000165[0] 1763350388.511872 write 0x0fb0cc thru 0x108ddc (0xdd11 bytes) 000166[1] 1763350388.554898 mapread 0x34e684 thru 0x35bf8f (0xd90c bytes) 000170[1] 1763350388.793939 write 0x03185e thru 0x03fae5 (0xe288 bytes) 000171[1] 1763350388.819440 read 0x2abb5c thru 0x2b96ca (0xdb6f bytes) 000172[1] 1763350388.856226 write 0x6a393a thru 0x6b2a74 (0xf13b bytes) 000173[1] 1763350388.868996 trunc from 0x9674cd to 0x0307e8 (0x936ce4 bytes) 000180[1] 1763350388.995124 write 0x0cbe0d thru 0x0d42bf (0x84b3 bytes) 000181[1] 1763350389.030934 read 0x0b5595 thru 0x0bcf12 (0x797e bytes) 000183[0] 1763350389.101135 trunc from 0x0d42c0 to 0x8c3d46 (0x7efa87 bytes) 000185[0] 1763350389.192082 read 0x135ae1 thru 0x13ac9f (0x51bf bytes) 000187[0] 1763350389.234811 read 0x7e0f04 thru 0x7ed0f4 (0xc1f1 bytes) 000188[0] 1763350389.248924 write 0x36567e thru 0x36f9c0 (0xa343 bytes) 000192[0] 1763350389.313783 write 0x329c23 thru 0x337f71 (0xe34f bytes) 000193[0] 1763350389.343903 write 0x999b09 thru 0x99f7b2 (0x5caa bytes) 000195[0] 1763350389.396249 mapwrite 0x23dcc1 thru 0x243174 (0x54b4 bytes) 000196[1] 1763350389.487621 mapread 0x1bf137 thru 0x1cc54c (0xd416 bytes) 000198[0] 1763350389.627142 mapread 0x3f5e7d thru 0x3fb199 (0x531d bytes) 000200[0] 1763350389.670267 mapwrite 0x3dadf2 thru 0x3de6ed (0x38fc bytes) 000204[1] 1763350389.761308 mapwrite 0x1da68c thru 0x1db1ff (0xb74 bytes) 000205[0] 1763350389.863669 trunc from 0x99f7b3 to 0x350e20 (0x64e992 bytes) 000207[0] 1763350389.933452 trunc from 0x350e21 to 0x62f15e (0x2de33e bytes) 000209[0] 1763350390.009098 write 0x87641a thru 0x884d60 (0xe947 bytes) 000211[1] 1763350390.040261 mapwrite 0x87ad66 thru 0x887bd9 (0xce74 bytes) 000213[0] 1763350390.384952 mapwrite 0x2f8956 thru 0x305968 (0xd013 bytes) truncating to largest ever: 0x96d6d9 000214[1] 1763350390.528266 trunc from 0x887bda to 0x96d6d8 (0xe5aff bytes) 000215[0] 1763350390.580386 trunc from 0x96d6d9 to 0x76a0b8 (0x203620 bytes) 000216[0] 1763350390.645931 mapwrite 0x5f3a10 thru 0x601bce (0xe1bf bytes) 000218[0] 1763350390.735693 mapread 0x39c331 thru 0x3a6573 (0xa243 bytes) 000220[1] 1763350390.779959 read 0x39fa9e thru 0x3a5fa6 (0x6509 bytes) 000221[0] 1763350390.813267 read 0x271725 thru 0x28140d (0xfce9 bytes) 000223[1] 1763350390.839575 trunc from 0x76a0b9 to 0x063346 (0x706d72 bytes) 000226[1] 1763350390.955042 mapread 0x03f65c thru 0x03fd92 (0x737 bytes) 000230[1] 1763350391.026705 read 0x032b0e thru 0x03364c (0xb3f bytes) 000231[0] 1763350391.060805 read 0x0369ed thru 0x03ccda (0x62ee bytes) 000233[0] 1763350391.149122 mapread 0x031a15 thru 0x033455 (0x1a41 bytes) 000234[0] 1763350391.201270 mapread 0x048b3d thru 0x057622 (0xeae6 bytes) 000238[0] 1763350391.275398 mapread 0x025c7e thru 0x02d973 (0x7cf6 bytes) 000240[0] 1763350391.299067 write 0x84058e thru 0x840f1c (0x98f bytes) 000242[0] 1763350391.346043 read 0x3d94ba thru 0x3e391c (0xa463 bytes) 000245[1] 1763350391.406895 mapread 0x69d2c2 thru 0x6a26e4 (0x5423 bytes) truncating to largest ever: 0x98c442 000249[1] 1763350391.536358 trunc from 0x840f1d to 0x98c441 (0x14b525 bytes) 000250[1] 1763350391.601863 mapread 0x7bffe7 thru 0x7cc2c5 (0xc2df bytes) 000251[1] 1763350391.744703 trunc from 0x98c442 to 0x60ce0a (0x37f637 bytes) 000253[1] 1763350391.816635 write 0x914e86 thru 0x92272a (0xd8a5 bytes) 000254[0] 1763350391.824205 mapread 0x6c31bb thru 0x6c79fb (0x4841 bytes) 000255[1] 1763350391.931472 mapwrite 0x570709 thru 0x57e408 (0xdd00 bytes) 000256[1] 1763350392.072284 mapread 0x79cc28 thru 0x7a2360 (0x5739 bytes) 000258[1] 1763350392.191388 write 0x218508 thru 0x21b413 (0x2f0c bytes) 000262[0] 1763350392.255336 mapread 0x44c82c thru 0x450332 (0x3b07 bytes) 000265[1] 1763350392.345543 read 0x064ca1 thru 0x070d9e (0xc0fe bytes) 000266[1] 1763350392.349029 mapread 0x28b8cc thru 0x290fbb (0x56f0 bytes) 000270[1] 1763350392.369381 write 0x18b5d2 thru 0x192423 (0x6e52 bytes) 000271[0] 1763350392.370365 mapread 0x5fc602 thru 0x606a4a (0xa449 bytes) 000275[0] 1763350392.427346 mapwrite 0x12c231 thru 0x13229c (0x606c bytes) 000277[0] 1763350392.556080 mapread 0x1dcb29 thru 0x1dea68 (0x1f40 bytes) 000279[0] 1763350392.569104 write 0x01007f thru 0x01d670 (0xd5f2 bytes) 000280[0] 1763350392.605082 mapwrite 0x942a19 thru 0x942f1f (0x507 bytes) 000283[1] 1763350392.727586 mapread 0x47a451 thru 0x483570 (0x9120 bytes) 000285[0] 1763350392.812689 mapwrite 0x5c8c3f thru 0x5cd03e (0x4400 bytes) 000286[1] 1763350392.952447 trunc from 0x942f20 to 0x098270 (0x8aacaf bytes) 000287[1] 1763350393.049052 read 0x087758 thru 0x08fd2e (0x85d7 bytes) 000288[1] 1763350393.073573 trunc from 0x098271 to 0x22afd1 (0x192d61 bytes) 000291[1] 1763350393.096375 mapwrite 0x5e85b6 thru 0x5ec4fe (0x3f49 bytes) 000292[0] 1763350393.262374 mapwrite 0x05248b thru 0x053f6e (0x1ae4 bytes) 000293[0] 1763350393.382981 trunc from 0x5ec4ff to 0x4f7747 (0xf4db7 bytes) 000297[0] 1763350393.464571 mapread 0x414316 thru 0x41cb65 (0x8850 bytes) 000300[1] 1763350393.498705 mapwrite 0x9906d3 thru 0x9911eb (0xb19 bytes) 000302[0] 1763350393.682215 write 0x1f056b thru 0x1f8906 (0x839c bytes) 000303[0] 1763350393.716096 mapread 0x7b54c3 thru 0x7bf204 (0x9d42 bytes) 000304[0] 1763350393.767963 mapwrite 0x7daf05 thru 0x7dbfcc (0x10c8 bytes) 000307[1] 1763350393.849255 read 0x52058b thru 0x52de4a (0xd8c0 bytes) 000309[0] 1763350393.886821 trunc from 0x9911ec to 0x2a8cfe (0x6e84ed bytes) 000310[1] 1763350393.928829 mapread 0x24583a thru 0x24a23f (0x4a06 bytes) 000311[1] 1763350394.010592 mapwrite 0x502616 thru 0x50cb18 (0xa503 bytes) 000312[1] 1763350394.147246 trunc from 0x50cb19 to 0x5953c3 (0x888ab bytes) 000314[1] 1763350394.164513 read 0x3b3c25 thru 0x3bd2f6 (0x96d2 bytes) 000315[0] 1763350394.172099 read 0x38a9b0 thru 0x396999 (0xbfea bytes) 000317[1] 1763350394.273957 trunc from 0x5953c4 to 0x881d08 (0x2ec945 bytes) 000319[0] 1763350394.306691 mapread 0x26c7cc thru 0x26f8d0 (0x3105 bytes) 000320[0] 1763350394.363407 mapwrite 0x3c0eff thru 0x3c8b59 (0x7c5b bytes) 000321[1] 1763350394.501682 trunc from 0x881d09 to 0x12d0ca (0x754c3e bytes) 000322[1] 1763350394.551246 mapread 0x0b6bf7 thru 0x0b9a2e (0x2e38 bytes) 000324[0] 1763350394.629491 mapread 0x0d6ca8 thru 0x0d75f9 (0x952 bytes) 000326[0] 1763350394.715119 read 0x0e12eb thru 0x0edc1e (0xc934 bytes) 000328[0] 1763350394.793378 trunc from 0x12d0cb to 0x6e2da3 (0x5b5cd9 bytes) 000329[0] 1763350394.877574 read 0x1be925 thru 0x1bedde (0x4ba bytes) 000330[0] 1763350394.965141 mapread 0x60f2df thru 0x61a4db (0xb1fd bytes) 000332[0] 1763350395.006029 mapread 0x3f1bad thru 0x3f1c39 (0x8d bytes) 000334[1] 1763350395.038651 write 0x597c62 thru 0x59a1a0 (0x253f bytes) 000335[0] 1763350395.119834 write 0x7ab1ca thru 0x7b6beb (0xba22 bytes) 000336[0] 1763350395.198530 write 0x3e6c25 thru 0x3f4ecc (0xe2a8 bytes) 000337[1] 1763350395.227613 read 0x6e9cbe thru 0x6f3a9f (0x9de2 bytes) 000339[0] 1763350395.283267 mapread 0x1d9588 thru 0x1e7a51 (0xe4ca bytes) 000342[1] 1763350395.366192 write 0x065f43 thru 0x072675 (0xc733 bytes) 000345[0] 1763350395.453217 read 0x53f76d thru 0x54e33b (0xebcf bytes) 000346[0] 1763350395.466104 read 0x3e847c thru 0x3f409a (0xbc1f bytes) 000347[0] 1763350395.495076 read 0x5824fa thru 0x58695b (0x4462 bytes) 000351[0] 1763350395.515882 mapwrite 0x7238b9 thru 0x729420 (0x5b68 bytes) 000353[0] 1763350395.665151 mapread 0x138120 thru 0x13f6ff (0x75e0 bytes) 000354[1] 1763350395.715851 trunc from 0x7b6bec to 0x394128 (0x422ac3 bytes) 000355[0] 1763350395.792941 trunc from 0x394129 to 0x506ddb (0x172cb3 bytes) 000356[0] 1763350395.862640 write 0x1475f1 thru 0x153dca (0xc7da bytes) 000357[0] 1763350395.903660 mapwrite 0x01a933 thru 0x024cce (0xa39c bytes) 000359[0] 1763350396.066211 trunc from 0x506ddc to 0x2023f5 (0x3049e6 bytes) 000360[1] 1763350396.094862 read 0x1d1512 thru 0x1dbd46 (0xa835 bytes) 000364[1] 1763350396.237452 mapwrite 0x75b5e6 thru 0x76905e (0xda79 bytes) 000366[1] 1763350396.372844 mapwrite 0x9f7341 thru 0x9fffff (0x8cbf bytes) 000367[0] 1763350396.528961 mapread 0x8fa736 thru 0x8fb306 (0xbd1 bytes) 000369[0] 1763350396.654143 mapwrite 0x0a972f thru 0x0abc6f (0x2541 bytes) 000372[0] 1763350396.948395 mapread 0x3debec thru 0x3edba6 (0xefbb bytes) 000373[1] 1763350397.021447 read 0x7bee39 thru 0x7cb06c (0xc234 bytes) 000376[0] 1763350397.134129 write 0x401d32 thru 0x4076e8 (0x59b7 bytes) 000378[0] 1763350397.265587 read 0x830035 thru 0x830afe (0xaca bytes) 000379[0] 1763350397.299934 mapread 0x82bdba thru 0x82e168 (0x23af bytes) 000380[0] 1763350397.401148 write 0x924927 thru 0x926585 (0x1c5f bytes) 000384[0] 1763350397.422256 write 0x129846 thru 0x12a267 (0xa22 bytes) 000385[1] 1763350397.448181 mapwrite 0x1df11b thru 0x1ecf37 (0xde1d bytes) 000388[1] 1763350397.646858 mapwrite 0x1ca2b8 thru 0x1cc64a (0x2393 bytes) 000389[0] 1763350397.725337 mapread 0x10ab4d thru 0x10faa3 (0x4f57 bytes) 000390[0] 1763350397.806175 write 0x2a2d1b thru 0x2b0ecb (0xe1b1 bytes) 000391[1] 1763350397.866680 write 0x35678f thru 0x364020 (0xd892 bytes) truncating to largest ever: 0x9add4c 000393[1] 1763350397.951829 trunc from 0xa00000 to 0x9add4b (0x522b4 bytes) 000395[0] 1763350398.041587 mapwrite 0x3f1ebf thru 0x3f53d1 (0x3513 bytes) 000396[1] 1763350398.211909 read 0x7e04da thru 0x7e0710 (0x237 bytes) 000398[1] 1763350398.264373 mapread 0x7ba06e thru 0x7c0e72 (0x6e05 bytes) 000400[0] 1763350398.339954 read 0x0cf3d4 thru 0x0d4ebf (0x5aec bytes) 000402[0] 1763350398.379395 write 0x3609ef thru 0x366ea9 (0x64bb bytes) 000403[1] 1763350398.408579 mapread 0x28c793 thru 0x297fb4 (0xb822 bytes) 000404[1] 1763350398.503162 write 0x4a34ac thru 0x4a3992 (0x4e7 bytes) 000407[1] 1763350398.576687 trunc from 0x9add4c to 0x98a721 (0x2362a bytes) 000408[0] 1763350398.672214 trunc from 0x98a722 to 0x401637 (0x5890ea bytes) 000409[1] 1763350398.742611 mapread 0x017c5e thru 0x025a66 (0xde09 bytes) truncating to largest ever: 0x9dbeca 000411[1] 1763350398.834081 trunc from 0x401638 to 0x9dbec9 (0x5da892 bytes) 000412[1] 1763350398.892987 mapwrite 0x34c6bf thru 0x34f567 (0x2ea9 bytes) 000413[1] 1763350398.960908 read 0x94ac83 thru 0x95a743 (0xfac1 bytes) 000416[1] 1763350399.057478 read 0x0f2b5b thru 0x0ff567 (0xca0d bytes) 000419[1] 1763350399.138520 read 0x3a3a42 thru 0x3aab79 (0x7138 bytes) truncating to largest ever: 0x9e82bf 000423[1] 1763350399.239716 trunc from 0x9dbeca to 0x9e82be (0xc3f5 bytes) 000425[1] 1763350399.323772 mapread 0x03eeb4 thru 0x04020b (0x1358 bytes) 000428[1] 1763350399.405096 trunc from 0x9e82bf to 0x0ff8f9 (0x8e89c5 bytes) 000429[0] 1763350399.435498 read 0x00f06e thru 0x01ef9b (0xff2e bytes) 000430[1] 1763350399.491242 write 0x83510f thru 0x839d59 (0x4c4b bytes) 000432[1] 1763350399.539205 mapwrite 0x960990 thru 0x969d9a (0x940b bytes) 000436[0] 1763350399.690006 trunc from 0x969d9b to 0x079da8 (0x8efff2 bytes) 000439[0] 1763350399.779210 mapread 0x0410ab thru 0x04836d (0x72c3 bytes) 000442[1] 1763350399.876318 write 0x7d5a15 thru 0x7da9f6 (0x4fe2 bytes) 000443[0] 1763350399.933486 mapwrite 0x891251 thru 0x893406 (0x21b6 bytes) 000444[1] 1763350400.181101 trunc from 0x893407 to 0x905bea (0x727e4 bytes) 000446[1] 1763350400.285955 write 0x3f1207 thru 0x3f139b (0x195 bytes) 000447[1] 1763350400.316414 mapread 0x0dbf5e thru 0x0e471c (0x87bf bytes) 000449[1] 1763350400.338974 mapwrite 0x0698c5 thru 0x074ad5 (0xb211 bytes) 000450[1] 1763350400.431193 trunc from 0x905beb to 0x8d3172 (0x32a78 bytes) 000454[1] 1763350400.484233 mapwrite 0x019082 thru 0x027ad9 (0xea58 bytes) 000455[1] 1763350400.559232 mapwrite 0x3ad35c thru 0x3af234 (0x1ed9 bytes) 000456[1] 1763350400.752137 read 0x2c28d6 thru 0x2d0f2d (0xe658 bytes) 000462[1] 1763350400.877916 write 0x2936e4 thru 0x297d42 (0x465f bytes) 000463[0] 1763350400.924045 write 0x835546 thru 0x836a58 (0x1513 bytes) 000464[0] 1763350400.992849 read 0x82c489 thru 0x82e8bb (0x2433 bytes) 000471[0] 1763350401.126711 trunc from 0x8d3173 to 0x8aca37 (0x2673b bytes) 000474[1] 1763350401.249495 mapwrite 0x58222a thru 0x58bedf (0x9cb6 bytes) 000475[0] 1763350401.396157 mapwrite 0x743568 thru 0x745869 (0x2302 bytes) 000476[0] 1763350401.504160 mapread 0x102878 thru 0x11184c (0xefd5 bytes) 000479[0] 1763350401.562620 mapwrite 0x06e91d thru 0x07917b (0xa85f bytes) 000480[0] 1763350401.677575 trunc from 0x8aca38 to 0x891fe1 (0x1aa56 bytes) 000482[0] 1763350401.743742 write 0x48b81a thru 0x497e8b (0xc672 bytes) 000483[0] 1763350401.759883 read 0x142ddc thru 0x146514 (0x3739 bytes) 000486[0] 1763350401.771264 trunc from 0x891fe2 to 0x2b33a8 (0x5dec39 bytes) 000487[0] 1763350401.840555 mapwrite 0x2a71a6 thru 0x2aad74 (0x3bcf bytes) 000490[0] 1763350402.019326 mapwrite 0x3f9471 thru 0x3ffb5c (0x66ec bytes) 000493[0] 1763350402.205255 write 0x471fe3 thru 0x478d04 (0x6d22 bytes) 000494[1] 1763350402.253391 write 0x29715b thru 0x297972 (0x818 bytes) 000495[1] 1763350402.288865 trunc from 0x478d05 to 0x5e1749 (0x168a45 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 402 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000008[1] 1763350403.100363 mapwrite 0x980000 thru 0x98ea6a (0xea6b bytes) truncating to largest ever: 0x770fb8 000010[1] 1763350403.255529 trunc from 0x98ea6b to 0x770fb7 (0x21dab3 bytes) truncating to largest ever: 0x7d5957 000011[1] 1763350403.292859 trunc from 0x770fb8 to 0x7d5956 (0x6499f bytes) 000013[1] 1763350403.378133 write_OD 0x9a9000 thru 0x9b6fff (0xe000 bytes) 000014[1] 1763350403.499018 write_OD 0x523000 thru 0x52dfff (0xb000 bytes) 000015[0] 1763350403.578000 mapwrite 0x307000 thru 0x30e8a1 (0x78a2 bytes) 000016[0] 1763350403.701109 mapwrite 0x3a4000 thru 0x3b2370 (0xe371 bytes) 000018[1] 1763350403.833228 write_OD 0x6b7000 thru 0x6b7fff (0x1000 bytes) 000019[0] 1763350403.892570 trunc from 0x9b7000 to 0x77671f (0x2408e0 bytes) truncating to largest ever: 0x88ec8f 000022[1] 1763350403.950118 trunc from 0x776720 to 0x88ec8e (0x11856f bytes) 000025[0] 1763350404.007118 mapwrite 0x580000 thru 0x582e11 (0x2e12 bytes) 000028[1] 1763350404.145179 read_OD 0x780000 thru 0x78afff (0xb000 bytes) 000030[0] 1763350404.200490 trunc from 0x88ec8f to 0x725552 (0x16973c bytes) 000036[0] 1763350404.333309 mapwrite 0x656000 thru 0x656222 (0x223 bytes) 000040[1] 1763350404.423063 mapwrite 0x5f6000 thru 0x5f78fe (0x18ff bytes) 000042[1] 1763350404.499066 mapwrite 0x0bf000 thru 0x0c87bb (0x97bc bytes) 000043[1] 1763350404.562651 write_OD 0x433000 thru 0x43bfff (0x9000 bytes) 000046[1] 1763350404.615544 mapwrite 0x26d000 thru 0x2786ea (0xb6eb bytes) 000047[1] 1763350404.695160 write_OD 0x920000 thru 0x92dfff (0xe000 bytes) 000048[0] 1763350404.744716 mapwrite 0x948000 thru 0x94a971 (0x2972 bytes) 000050[0] 1763350404.858429 write_OD 0x6d4000 thru 0x6d7fff (0x4000 bytes) 000051[0] 1763350404.942964 write_OD 0x6a6000 thru 0x6a7fff (0x2000 bytes) 000052[1] 1763350404.999684 read_OD 0x2da000 thru 0x2e7fff (0xe000 bytes) 000054[1] 1763350405.018278 mapwrite 0x900000 thru 0x90d353 (0xd354 bytes) 000055[0] 1763350405.112545 read_OD 0x007000 thru 0x00cfff (0x6000 bytes) 000062[0] 1763350405.138401 write_OD 0x5cd000 thru 0x5d1fff (0x5000 bytes) 000064[1] 1763350405.198713 trunc from 0x94a972 to 0x0b40fc (0x896875 bytes) 000066[0] 1763350405.265357 write_OD 0x2eb000 thru 0x2f6fff (0xc000 bytes) 000067[1] 1763350405.310966 write_OD 0x344000 thru 0x346fff (0x3000 bytes) 000068[0] 1763350405.356519 mapwrite 0x09b000 thru 0x0a7c43 (0xcc44 bytes) 000069[1] 1763350405.446260 mapwrite 0x2b7000 thru 0x2c5122 (0xe123 bytes) 000070[1] 1763350405.535887 trunc from 0x347000 to 0x1802d3 (0x1c6d2c bytes) 000071[1] 1763350405.566597 read_OD 0x0f2000 thru 0x0fafff (0x9000 bytes) truncating to largest ever: 0x973b21 000074[1] 1763350405.596034 trunc from 0x1802d4 to 0x973b20 (0x7f384d bytes) 000078[1] 1763350405.653582 trunc from 0x973b21 to 0x55bd97 (0x417d89 bytes) 000080[1] 1763350405.696445 write_OD 0x6d7000 thru 0x6d9fff (0x3000 bytes) 000081[1] 1763350405.746833 write_OD 0x8b3000 thru 0x8bcfff (0xa000 bytes) 000084[0] 1763350405.803092 read_OD 0x580000 thru 0x580fff (0x1000 bytes) 000085[1] 1763350405.807710 trunc from 0x8bd000 to 0x1f4bcd (0x6c8432 bytes) 000087[0] 1763350405.871846 write_OD 0x855000 thru 0x855fff (0x1000 bytes) 000089[0] 1763350405.931062 mapread 0x7c3000 thru 0x7c360a (0x60b bytes) 000091[1] 1763350405.936403 read_OD 0x4cc000 thru 0x4cdfff (0x2000 bytes) 000092[1] 1763350405.941230 trunc from 0x856000 to 0x5a1f45 (0x2b40ba bytes) 000093[1] 1763350405.996063 read_OD 0x2bb000 thru 0x2bbfff (0x1000 bytes) 000095[1] 1763350406.011274 mapread 0x1be000 thru 0x1beb5e (0xb5f bytes) 000096[0] 1763350406.024266 trunc from 0x5a1f46 to 0x665252 (0xc330d bytes) 000098[1] 1763350406.076118 read_OD 0x203000 thru 0x20dfff (0xb000 bytes) 000101[1] 1763350406.108082 read_OD 0x4e5000 thru 0x4f0fff (0xc000 bytes) 000104[0] 1763350406.121558 write_OD 0x54c000 thru 0x553fff (0x8000 bytes) 000105[1] 1763350406.168691 trunc from 0x665253 to 0x723f48 (0xbecf6 bytes) 000106[1] 1763350406.201313 mapread 0x576000 thru 0x58514a (0xf14b bytes) 000109[0] 1763350406.267453 mapwrite 0x7e7000 thru 0x7ebf13 (0x4f14 bytes) 000110[1] 1763350406.393277 trunc from 0x7ebf14 to 0x64eeee (0x19d025 bytes) 000111[0] 1763350406.415817 trunc from 0x64eeef to 0x6b9a3f (0x6ab51 bytes) 000112[1] 1763350406.457976 trunc from 0x6b9a40 to 0x2b2a36 (0x407009 bytes) 000113[0] 1763350406.493404 mapwrite 0x4cf000 thru 0x4d8afd (0x9afe bytes) 000116[1] 1763350406.601456 write_OD 0x268000 thru 0x276fff (0xf000 bytes) 000118[0] 1763350406.648259 mapread 0x36b000 thru 0x36f8f2 (0x48f3 bytes) 000122[0] 1763350406.671911 mapwrite 0x220000 thru 0x22757d (0x757e bytes) 000123[1] 1763350406.753072 mapwrite 0x0b8000 thru 0x0c5dd1 (0xddd2 bytes) 000124[1] 1763350406.881266 mapwrite 0x042000 thru 0x044db1 (0x2db2 bytes) 000125[0] 1763350406.945241 trunc from 0x4d8afe to 0x32d709 (0x1ab3f4 bytes) 000126[1] 1763350406.996170 read_OD 0x252000 thru 0x258fff (0x7000 bytes) 000127[0] 1763350407.019365 write_OD 0x5c8000 thru 0x5c9fff (0x2000 bytes) 000130[1] 1763350407.064031 mapread 0x14f000 thru 0x15e946 (0xf947 bytes) 000132[0] 1763350407.144629 write_OD 0x8d3000 thru 0x8d7fff (0x5000 bytes) 000136[1] 1763350407.227172 mapread 0x371000 thru 0x374e02 (0x3e03 bytes) 000138[1] 1763350407.248485 trunc from 0x8d8000 to 0x0d95cb (0x7fea34 bytes) 000139[1] 1763350407.294837 mapwrite 0x76d000 thru 0x76ee6d (0x1e6e bytes) 000144[1] 1763350407.493150 trunc from 0x76ee6e to 0x5d80eb (0x196d82 bytes) 000147[1] 1763350407.547731 read_OD 0x3a5000 thru 0x3aafff (0x6000 bytes) 000148[0] 1763350407.577572 mapread 0x1bf000 thru 0x1bf865 (0x866 bytes) 000151[0] 1763350407.596028 read_OD 0x28d000 thru 0x28dfff (0x1000 bytes) 000153[0] 1763350407.607333 mapread 0x18b000 thru 0x18fc3f (0x4c40 bytes) 000154[0] 1763350407.634983 mapread 0x3a0000 thru 0x3a13c0 (0x13c1 bytes) 000156[1] 1763350407.649109 write_OD 0x254000 thru 0x259fff (0x6000 bytes) 000157[1] 1763350407.716324 trunc from 0x5d80ec to 0x54a04c (0x8e09f bytes) 000159[0] 1763350407.799081 mapread 0x448000 thru 0x453c2c (0xbc2d bytes) 000161[1] 1763350407.845463 read_OD 0x443000 thru 0x450fff (0xe000 bytes) 000162[0] 1763350407.873737 mapread 0x450000 thru 0x459a44 (0x9a45 bytes) 000163[0] 1763350407.877088 mapwrite 0x150000 thru 0x15b8be (0xb8bf bytes) 000164[0] 1763350408.003023 mapread 0x275000 thru 0x279e51 (0x4e52 bytes) 000167[0] 1763350408.075141 mapread 0x4fc000 thru 0x5004bc (0x44bd bytes) 000169[0] 1763350408.123618 trunc from 0x54a04d to 0x8f1901 (0x3a78b5 bytes) 000170[1] 1763350408.178387 mapread 0x162000 thru 0x170bc7 (0xebc8 bytes) 000173[1] 1763350408.248523 trunc from 0x8f1902 to 0x67559f (0x27c362 bytes) 000175[0] 1763350408.327160 mapwrite 0x111000 thru 0x11b723 (0xa724 bytes) 000176[0] 1763350408.468171 mapread 0x176000 thru 0x1839a5 (0xd9a6 bytes) 000180[1] 1763350408.541407 read_OD 0x0e2000 thru 0x0ebfff (0xa000 bytes) 000183[0] 1763350408.585347 read_OD 0x4c4000 thru 0x4c9fff (0x6000 bytes) 000184[1] 1763350408.594011 mapread 0x499000 thru 0x4a8ced (0xfcee bytes) 000185[1] 1763350408.628535 mapread 0x0cd000 thru 0x0d29b6 (0x59b7 bytes) 000187[0] 1763350408.658863 trunc from 0x6755a0 to 0x649e82 (0x2b71d bytes) 000188[1] 1763350408.710223 write_OD 0x877000 thru 0x87dfff (0x7000 bytes) 000190[1] 1763350408.817014 write_OD 0x39f000 thru 0x3aafff (0xc000 bytes) 000192[1] 1763350408.907522 read_OD 0x719000 thru 0x71efff (0x6000 bytes) 000194[1] 1763350408.928615 mapwrite 0x739000 thru 0x739359 (0x35a bytes) 000195[0] 1763350409.034961 mapwrite 0x616000 thru 0x618b24 (0x2b25 bytes) 000197[0] 1763350409.154953 mapwrite 0x0de000 thru 0x0e302e (0x502f bytes) 000198[0] 1763350409.231887 write_OD 0x408000 thru 0x40afff (0x3000 bytes) 000199[1] 1763350409.314165 read_OD 0x7a8000 thru 0x7b1fff (0xa000 bytes) 000200[0] 1763350409.327430 mapwrite 0x5da000 thru 0x5dcdbf (0x2dc0 bytes) 000201[0] 1763350409.413008 mapread 0x602000 thru 0x60595c (0x395d bytes) 000202[0] 1763350409.447064 mapread 0x12b000 thru 0x13a03a (0xf03b bytes) 000204[1] 1763350409.515069 read_OD 0x30e000 thru 0x31bfff (0xe000 bytes) 000205[1] 1763350409.554507 write_OD 0x119000 thru 0x124fff (0xc000 bytes) 000206[1] 1763350409.630093 mapread 0x026000 thru 0x033b6f (0xdb70 bytes) 000207[0] 1763350409.673386 mapwrite 0x614000 thru 0x618c05 (0x4c06 bytes) 000209[1] 1763350409.775859 trunc from 0x87e000 to 0x75fce3 (0x11e31c bytes) truncating to largest ever: 0x9fded1 000212[0] 1763350409.811352 trunc from 0x75fce4 to 0x9fded0 (0x29e1ed bytes) 000213[0] 1763350409.843700 mapwrite 0x477000 thru 0x4805fc (0x95fd bytes) 000214[0] 1763350409.923438 mapwrite 0x10e000 thru 0x11020d (0x220e bytes) 000215[0] 1763350409.975185 mapread 0x7d2000 thru 0x7dfaae (0xdaaf bytes) 000216[1] 1763350410.028555 trunc from 0x9fded1 to 0x98ed67 (0x6f169 bytes) 000220[0] 1763350410.066945 read_OD 0x09c000 thru 0x0aafff (0xf000 bytes) 000223[1] 1763350410.102684 trunc from 0x98ed68 to 0x0bfe34 (0x8cef33 bytes) 000226[1] 1763350410.234691 mapwrite 0x42d000 thru 0x439ea9 (0xceaa bytes) 000228[1] 1763350410.327421 trunc from 0x439eaa to 0x1b468d (0x28581c bytes) 000229[0] 1763350410.349101 read_OD 0x0aa000 thru 0x0b6fff (0xd000 bytes) 000230[1] 1763350410.371602 mapread 0x171000 thru 0x17a5e6 (0x95e7 bytes) 000233[1] 1763350410.393171 mapwrite 0x2d9000 thru 0x2dac53 (0x1c54 bytes) 000237[1] 1763350410.485969 read_OD 0x1b3000 thru 0x1b9fff (0x7000 bytes) 000238[0] 1763350410.503582 mapwrite 0x886000 thru 0x889c9b (0x3c9c bytes) 000240[1] 1763350410.806539 trunc from 0x889c9c to 0x60a28f (0x27fa0c bytes) 000241[1] 1763350410.863941 write_OD 0x582000 thru 0x58afff (0x9000 bytes) 000244[1] 1763350410.915606 write_OD 0x121000 thru 0x128fff (0x8000 bytes) 000245[0] 1763350410.960122 mapread 0x5ad000 thru 0x5b83e1 (0xb3e2 bytes) 000246[1] 1763350411.040125 mapwrite 0x748000 thru 0x7574bb (0xf4bc bytes) 000247[0] 1763350411.324570 write_OD 0x0de000 thru 0x0e4fff (0x7000 bytes) 000250[0] 1763350411.391250 write_OD 0x8e8000 thru 0x8ecfff (0x5000 bytes) 000251[1] 1763350411.439307 mapread 0x538000 thru 0x544668 (0xc669 bytes) 000253[1] 1763350411.499134 trunc from 0x8ed000 to 0x9c14b4 (0xd44b5 bytes) 000255[1] 1763350411.549943 read_OD 0x233000 thru 0x23afff (0x8000 bytes) 000256[1] 1763350411.567871 trunc from 0x9c14b5 to 0x466962 (0x55ab52 bytes) 000258[0] 1763350411.622955 read_OD 0x41e000 thru 0x423fff (0x6000 bytes) 000260[1] 1763350411.652844 mapread 0x17f000 thru 0x184a51 (0x5a52 bytes) 000261[1] 1763350411.698031 write_OD 0x311000 thru 0x313fff (0x3000 bytes) 000262[1] 1763350411.805727 read_OD 0x2a0000 thru 0x2aafff (0xb000 bytes) 000263[1] 1763350411.829119 mapwrite 0x1e9000 thru 0x1f4e93 (0xbe94 bytes) 000266[0] 1763350411.932323 read_OD 0x114000 thru 0x118fff (0x5000 bytes) 000269[1] 1763350411.983481 read_OD 0x25e000 thru 0x25efff (0x1000 bytes) 000271[1] 1763350411.991484 mapwrite 0x9f2000 thru 0x9fd323 (0xb324 bytes) 000276[1] 1763350412.135978 mapread 0x7d2000 thru 0x7d483d (0x283e bytes) 000278[1] 1763350412.165649 write_OD 0x14b000 thru 0x156fff (0xc000 bytes) 000279[0] 1763350412.212225 mapwrite 0x5ef000 thru 0x5fc7c2 (0xd7c3 bytes) 000280[1] 1763350412.298858 write_OD 0x963000 thru 0x964fff (0x2000 bytes) 000282[0] 1763350412.346205 write_OD 0x7e2000 thru 0x7e6fff (0x5000 bytes) 000283[0] 1763350412.413636 mapread 0x598000 thru 0x598d71 (0xd72 bytes) 000284[0] 1763350412.424801 trunc from 0x9fd324 to 0x51998c (0x4e3997 bytes) 000288[1] 1763350412.505241 mapwrite 0x022000 thru 0x024c78 (0x2c79 bytes) 000289[1] 1763350412.638912 trunc from 0x51998d to 0x55f6bd (0x45d31 bytes) 000291[1] 1763350412.697100 read_OD 0x113000 thru 0x115fff (0x3000 bytes) 000293[1] 1763350412.724716 read_OD 0x12e000 thru 0x13cfff (0xf000 bytes) 000296[0] 1763350412.734889 mapread 0x4fa000 thru 0x500dba (0x6dbb bytes) 000297[0] 1763350412.762263 trunc from 0x55f6be to 0x70359b (0x1a3ede bytes) 000299[1] 1763350412.845987 trunc from 0x70359c to 0x67f343 (0x84258 bytes) 000303[0] 1763350412.901374 mapread 0x2c6000 thru 0x2c7f2b (0x1f2c bytes) 000308[0] 1763350412.964055 trunc from 0x67f344 to 0x0dc043 (0x5a3300 bytes) 000310[0] 1763350413.057008 read_OD 0x07e000 thru 0x08cfff (0xf000 bytes) 000311[1] 1763350413.084718 write_OD 0x5f8000 thru 0x600fff (0x9000 bytes) 000312[0] 1763350413.156067 write_OD 0x60f000 thru 0x61cfff (0xe000 bytes) 000313[0] 1763350413.205687 read_OD 0x2b9000 thru 0x2bbfff (0x3000 bytes) 000314[0] 1763350413.213933 read_OD 0x0b4000 thru 0x0bffff (0xc000 bytes) 000315[0] 1763350413.221145 read_OD 0x5a6000 thru 0x5a6fff (0x1000 bytes) 000316[1] 1763350413.225725 mapread 0x198000 thru 0x1a5f81 (0xdf82 bytes) 000320[1] 1763350413.250322 mapread 0x3ae000 thru 0x3af4eb (0x14ec bytes) 000321[0] 1763350413.260552 trunc from 0x61d000 to 0x9b6dbf (0x399dc0 bytes) 000323[0] 1763350413.293250 mapread 0x48e000 thru 0x49d09c (0xf09d bytes) 000330[1] 1763350413.386132 write_OD 0x8cf000 thru 0x8d6fff (0x8000 bytes) 000331[1] 1763350413.447699 mapread 0x699000 thru 0x69f286 (0x6287 bytes) 000332[1] 1763350413.483889 mapwrite 0x513000 thru 0x51796d (0x496e bytes) 000337[1] 1763350413.635598 trunc from 0x9b6dc0 to 0x771daf (0x245010 bytes) 000338[1] 1763350413.655246 mapwrite 0x163000 thru 0x163558 (0x559 bytes) 000339[0] 1763350413.716583 trunc from 0x771db0 to 0x79b868 (0x29ab9 bytes) 000345[0] 1763350413.785215 write_OD 0x6de000 thru 0x6e0fff (0x3000 bytes) 000346[1] 1763350413.831622 mapread 0x4a6000 thru 0x4b2d8a (0xcd8b bytes) 000347[1] 1763350413.861679 trunc from 0x79b869 to 0x524ee9 (0x27697f bytes) 000350[1] 1763350413.936455 mapread 0x1f1000 thru 0x1f6ca5 (0x5ca6 bytes) 000352[0] 1763350413.960166 read_OD 0x29a000 thru 0x2a7fff (0xe000 bytes) 000355[1] 1763350414.005250 write_OD 0x0fd000 thru 0x0fdfff (0x1000 bytes) 000357[0] 1763350414.055575 mapwrite 0x16c000 thru 0x1720b9 (0x60ba bytes) 000360[0] 1763350414.145248 trunc from 0x524eea to 0x9a4f80 (0x480097 bytes) 000361[1] 1763350414.170235 write_OD 0x025000 thru 0x02efff (0xa000 bytes) 000365[0] 1763350414.251694 trunc from 0x9a4f81 to 0x497d09 (0x50d277 bytes) 000368[0] 1763350414.307634 mapwrite 0x407000 thru 0x40c318 (0x5319 bytes) 000369[1] 1763350414.378266 write_OD 0x1db000 thru 0x1e0fff (0x6000 bytes) 000371[1] 1763350414.426236 read_OD 0x0bf000 thru 0x0c9fff (0xb000 bytes) 000372[1] 1763350414.432998 mapwrite 0x96d000 thru 0x96d510 (0x511 bytes) 000374[0] 1763350414.529643 trunc from 0x96d511 to 0x578b34 (0x3f49dc bytes) 000375[1] 1763350414.596322 mapread 0x012000 thru 0x01e22f (0xc230 bytes) 000377[1] 1763350414.708985 write_OD 0x1bc000 thru 0x1c0fff (0x5000 bytes) 000380[1] 1763350414.782521 trunc from 0x578b35 to 0x2aafd2 (0x2cdb62 bytes) 000381[0] 1763350414.833162 write_OD 0x259000 thru 0x264fff (0xc000 bytes) 000382[0] 1763350414.875476 trunc from 0x2aafd3 to 0x395f1e (0xeaf4c bytes) 000385[1] 1763350414.961190 mapwrite 0x60d000 thru 0x61579c (0x879d bytes) 000386[0] 1763350415.074337 trunc from 0x61579d to 0x665502 (0x4fd66 bytes) 000389[1] 1763350415.123938 write_OD 0x2d8000 thru 0x2e4fff (0xd000 bytes) 000392[1] 1763350415.178120 write_OD 0x392000 thru 0x399fff (0x8000 bytes) 000396[1] 1763350415.238535 mapwrite 0x21e000 thru 0x223e0a (0x5e0b bytes) 000397[1] 1763350415.318328 trunc from 0x665503 to 0x9e0959 (0x37b457 bytes) 000400[1] 1763350415.356124 mapwrite 0x6bc000 thru 0x6c64db (0xa4dc bytes) 000404[1] 1763350415.414849 trunc from 0x9e095a to 0x2eb719 (0x6f5240 bytes) 000405[0] 1763350415.431578 mapread 0x0a3000 thru 0x0a8cc8 (0x5cc9 bytes) 000407[1] 1763350415.479457 write_OD 0x765000 thru 0x766fff (0x2000 bytes) 000408[1] 1763350415.526970 read_OD 0x105000 thru 0x112fff (0xe000 bytes) 000409[0] 1763350415.534237 read_OD 0x49d000 thru 0x4a7fff (0xb000 bytes) 000410[1] 1763350415.544327 trunc from 0x767000 to 0x0ba079 (0x6acf86 bytes) 000411[1] 1763350415.588384 mapwrite 0x526000 thru 0x52cd88 (0x6d89 bytes) 000415[1] 1763350415.738046 mapread 0x27c000 thru 0x28285c (0x685d bytes) 000416[0] 1763350415.775264 mapread 0x184000 thru 0x192ac3 (0xeac4 bytes) 000417[0] 1763350415.842342 write_OD 0x521000 thru 0x525fff (0x5000 bytes) 000418[0] 1763350415.930165 read_OD 0x47c000 thru 0x48afff (0xf000 bytes) 000419[1] 1763350415.953146 read_OD 0x375000 thru 0x382fff (0xe000 bytes) 000420[0] 1763350415.981986 trunc from 0x52cd89 to 0x89590a (0x368b82 bytes) 000421[1] 1763350416.043110 mapwrite 0x37b000 thru 0x387ebc (0xcebd bytes) 000422[1] 1763350416.170104 trunc from 0x89590b to 0x9833ec (0xedae2 bytes) 000423[0] 1763350416.246309 read_OD 0x24d000 thru 0x24efff (0x2000 bytes) 000424[0] 1763350416.289516 read_OD 0x29a000 thru 0x2a5fff (0xc000 bytes) 000427[0] 1763350416.311340 read_OD 0x132000 thru 0x138fff (0x7000 bytes) 000428[1] 1763350416.321202 mapread 0x218000 thru 0x21b9fb (0x39fc bytes) 000429[0] 1763350416.347329 write_OD 0x6b3000 thru 0x6bdfff (0xb000 bytes) 000431[1] 1763350416.407480 mapwrite 0x08d000 thru 0x09a359 (0xd35a bytes) 000432[1] 1763350416.538417 trunc from 0x9833ed to 0x98a880 (0x7494 bytes) 000433[0] 1763350416.579648 trunc from 0x98a881 to 0x5bd24d (0x3cd633 bytes) 000437[1] 1763350416.646004 trunc from 0x5bd24e to 0x3df029 (0x1de224 bytes) 000439[1] 1763350416.684032 read_OD 0x148000 thru 0x150fff (0x9000 bytes) 000440[1] 1763350416.715795 write_OD 0x27c000 thru 0x27ffff (0x4000 bytes) 000445[1] 1763350416.774571 mapread 0x352000 thru 0x357e81 (0x5e82 bytes) 000446[0] 1763350416.840196 mapread 0x01c000 thru 0x022d4d (0x6d4e bytes) 000448[1] 1763350416.881021 read_OD 0x2aa000 thru 0x2b1fff (0x8000 bytes) 000450[0] 1763350416.894271 mapwrite 0x357000 thru 0x35c2c1 (0x52c2 bytes) 000452[0] 1763350417.023875 mapread 0x1ab000 thru 0x1b866c (0xd66d bytes) 000453[1] 1763350417.049889 mapread 0x11b000 thru 0x11b0b2 (0xb3 bytes) 000454[0] 1763350417.122495 mapwrite 0x882000 thru 0x885e30 (0x3e31 bytes) 000457[0] 1763350417.267947 read_OD 0x7fc000 thru 0x805fff (0xa000 bytes) 000458[1] 1763350417.296511 write_OD 0x061000 thru 0x06ffff (0xf000 bytes) 000460[1] 1763350417.360189 mapwrite 0x460000 thru 0x4645d5 (0x45d6 bytes) 000462[0] 1763350417.501654 write_OD 0x71c000 thru 0x72afff (0xf000 bytes) 000463[1] 1763350417.571940 mapwrite 0x53b000 thru 0x5410d4 (0x60d5 bytes) 000467[0] 1763350417.733534 mapwrite 0x9c1000 thru 0x9cf018 (0xe019 bytes) 000468[0] 1763350417.903574 mapwrite 0x396000 thru 0x3a43ac (0xe3ad bytes) 000470[0] 1763350418.053910 trunc from 0x9cf019 to 0x2e0a5b (0x6ee5bd bytes) 000471[0] 1763350418.092203 trunc from 0x2e0a5c to 0x5a7247 (0x2c67ec bytes) 000477[0] 1763350418.180613 mapread 0x426000 thru 0x42ab78 (0x4b79 bytes) 000479[0] 1763350418.224429 mapwrite 0x22c000 thru 0x22e03a (0x203b bytes) 000480[0] 1763350418.346957 read_OD 0x390000 thru 0x395fff (0x6000 bytes) 000481[0] 1763350418.383015 mapwrite 0x447000 thru 0x44b3a9 (0x43aa bytes) 000485[0] 1763350418.482718 read_OD 0x3fc000 thru 0x3fdfff (0x2000 bytes) 000487[0] 1763350418.502308 write_OD 0x794000 thru 0x79ffff (0xc000 bytes) 000488[1] 1763350418.549227 mapread 0x5fa000 thru 0x5fd912 (0x3913 bytes) 000490[0] 1763350418.588398 mapwrite 0x34f000 thru 0x34ff38 (0xf39 bytes) 000492[1] 1763350418.675806 mapread 0x137000 thru 0x1465a5 (0xf5a6 bytes) 000493[1] 1763350418.751544 read_OD 0x13c000 thru 0x148fff (0xd000 bytes) 000494[0] 1763350418.759192 mapwrite 0x24c000 thru 0x25b872 (0xf873 bytes) 000496[1] 1763350418.873183 write_OD 0x385000 thru 0x391fff (0xd000 bytes) 000497[0] 1763350418.929946 write_OD 0x54c000 thru 0x558fff (0xd000 bytes) 000500[1] 1763350419.009450 mapwrite 0x8a3000 thru 0x8b1ff2 (0xeff3 bytes) All operations completed A-OK! PASS 16a (46s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 22:33:43 (1763350423) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.581307 s, 14.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.406585 s, 20.6 MB/s Chance of close/open is 1 in 50 Seed set to 426 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1763350427.834424 mapwrite 0x790132 thru 0x79123c (0x110b bytes) truncating to largest ever: 0x674ea3 000003[0] 1763350427.988664 trunc from 0x79123d to 0x674ea2 (0x11c39a bytes) 000006[1] 1763350428.113794 mapread 0x432910 thru 0x433b41 (0x1232 bytes) 000008[1] 1763350428.185111 mapwrite 0x87f015 thru 0x87fbc2 (0xbae bytes) 000009[1] 1763350428.357022 mapread 0x0011af thru 0x0026a9 (0x14fb bytes) 000010[1] 1763350428.418922 mapread 0x1f8105 thru 0x1f94fa (0x13f6 bytes) 000016[1] 1763350428.654947 mapwrite 0x20eff1 thru 0x2100e6 (0x10f6 bytes) 000017[0] 1763350428.778155 mapwrite 0x2d1a3b thru 0x2d299b (0xf61 bytes) 000025[0] 1763350429.172816 mapread 0x648652 thru 0x648a57 (0x406 bytes) 000031[1] 1763350429.404002 trunc from 0x87fbc3 to 0x267485 (0x61873d bytes) 000037[0] 1763350429.649815 mapwrite 0x05f275 thru 0x060df8 (0x1b84 bytes) 000038[1] 1763350429.866100 read 0x1babb7 thru 0x1bbee1 (0x132b bytes) 000040[1] 1763350429.947922 read 0x0b2c5f thru 0x0b3ad0 (0xe72 bytes) 000041[1] 1763350430.011066 mapread 0x0f5e84 thru 0x0f6899 (0xa16 bytes) 000042[1] 1763350430.012001 mapwrite 0x898481 thru 0x89985e (0x13de bytes) 000043[1] 1763350430.263228 trunc from 0x89985f to 0x0d40c2 (0x7c579c bytes) 000044[0] 1763350430.302706 trunc from 0x0d40c3 to 0x23c408 (0x168346 bytes) 000045[1] 1763350430.443089 read 0x1accb4 thru 0x1adef9 (0x1246 bytes) 000048[1] 1763350430.560410 mapread 0x0f5776 thru 0x0f6dac (0x1637 bytes) 000052[1] 1763350430.570139 mapwrite 0x6d6a3e thru 0x6d6f07 (0x4ca bytes) 000055[1] 1763350430.873217 write 0x529203 thru 0x529516 (0x314 bytes) 000056[1] 1763350431.015985 write 0x27ec3e thru 0x27fd93 (0x1156 bytes) 000058[1] 1763350431.019447 write 0x48ff88 thru 0x491b85 (0x1bfe bytes) 000059[0] 1763350431.021421 mapread 0x239c69 thru 0x23a80c (0xba4 bytes) 000060[0] 1763350431.090287 mapwrite 0x897343 thru 0x899163 (0x1e21 bytes) 000064[0] 1763350431.382455 mapread 0x67dc7f thru 0x67f75c (0x1ade bytes) 000067[1] 1763350431.457205 mapread 0x7d3fb5 thru 0x7d5c1b (0x1c67 bytes) 000068[0] 1763350431.542143 mapwrite 0x059e1c thru 0x05ae4a (0x102f bytes) truncating to largest ever: 0x8812b1 000070[0] 1763350431.694953 trunc from 0x899164 to 0x8812b0 (0x17eb3 bytes) 000071[0] 1763350431.752448 mapread 0x3d65d1 thru 0x3d7fea (0x1a1a bytes) 000073[1] 1763350431.770214 write 0x85302f thru 0x853322 (0x2f4 bytes) 000074[1] 1763350431.817652 mapread 0x1aff08 thru 0x1b12ea (0x13e3 bytes) 000079[1] 1763350431.969064 read 0x5e5056 thru 0x5e6d57 (0x1d02 bytes) 000080[1] 1763350432.005432 mapwrite 0x31b50e thru 0x31cda0 (0x1893 bytes) 000081[1] 1763350432.094126 read 0x289bf9 thru 0x28ba37 (0x1e3f bytes) 000082[1] 1763350432.101702 trunc from 0x8812b1 to 0x3daac9 (0x4a67e7 bytes) 000083[1] 1763350432.142654 mapread 0x249d04 thru 0x24a7ab (0xaa8 bytes) 000084[0] 1763350432.188333 mapwrite 0x580322 thru 0x580a2c (0x70b bytes) 000085[1] 1763350432.304509 write 0x343047 thru 0x343cf0 (0xcaa bytes) 000086[1] 1763350432.340666 mapread 0x477554 thru 0x479262 (0x1d0f bytes) 000090[0] 1763350432.427487 mapwrite 0x7c5e4b thru 0x7c6c28 (0xdde bytes) 000095[0] 1763350432.683963 mapwrite 0x696684 thru 0x697204 (0xb81 bytes) 000096[0] 1763350432.743225 trunc from 0x7c6c29 to 0x45eeec (0x367d3c bytes) 000099[0] 1763350432.802929 read 0x197e85 thru 0x198f2a (0x10a6 bytes) 000101[0] 1763350432.899380 trunc from 0x45eeed to 0x62683b (0x1c794f bytes) 000103[0] 1763350432.989848 read 0x374ab0 thru 0x3751e4 (0x735 bytes) 000106[0] 1763350433.070963 trunc from 0x62683c to 0x1ec250 (0x43a5eb bytes) 000109[1] 1763350433.153238 mapwrite 0x3c3cd8 thru 0x3c5ac2 (0x1deb bytes) 000112[0] 1763350433.362085 trunc from 0x3c5ac3 to 0x38e3eb (0x376d7 bytes) 000113[1] 1763350433.460664 mapread 0x147d20 thru 0x149a34 (0x1d15 bytes) 000114[1] 1763350433.520540 mapread 0x1e79d2 thru 0x1e912b (0x175a bytes) 000116[0] 1763350433.574844 mapwrite 0x687fa8 thru 0x6881f0 (0x249 bytes) 000118[0] 1763350433.768283 mapwrite 0x57290c thru 0x573d82 (0x1477 bytes) 000122[1] 1763350433.902876 mapwrite 0x2c0d37 thru 0x2c11d8 (0x4a2 bytes) 000123[0] 1763350434.034818 trunc from 0x6881f1 to 0x7e10dc (0x158eec bytes) 000127[1] 1763350434.101596 mapwrite 0x4a1324 thru 0x4a1519 (0x1f6 bytes) 000128[1] 1763350434.215576 write 0x0a3f0d thru 0x0a4524 (0x618 bytes) 000129[0] 1763350434.234152 trunc from 0x7e10dd to 0x3b6ccc (0x42a410 bytes) 000133[0] 1763350434.335600 write 0x9f42b1 thru 0x9f4db3 (0xb03 bytes) 000134[0] 1763350434.371811 read 0x9453ab thru 0x946003 (0xc59 bytes) 000138[0] 1763350434.411249 trunc from 0x9f4db4 to 0x7892ae (0x26bb05 bytes) 000139[1] 1763350434.467858 trunc from 0x7892af to 0x856ce3 (0xcda35 bytes) 000140[0] 1763350434.513580 trunc from 0x856ce4 to 0x5f36b8 (0x26362b bytes) 000145[0] 1763350434.619829 write 0x6ac6bb thru 0x6acf26 (0x86c bytes) 000147[0] 1763350434.646335 trunc from 0x6acf27 to 0x06d3e7 (0x63fb3f bytes) 000151[0] 1763350434.756784 write 0x493cb3 thru 0x494fd9 (0x1327 bytes) 000152[1] 1763350434.761394 mapread 0x0dbf56 thru 0x0dca2e (0xad9 bytes) 000155[1] 1763350434.843616 read 0x0603bb thru 0x061522 (0x1168 bytes) 000156[1] 1763350434.874048 read 0x00cf08 thru 0x00e3cd (0x14c6 bytes) 000159[0] 1763350434.918627 write 0x5df530 thru 0x5e12f4 (0x1dc5 bytes) 000161[1] 1763350434.920920 read 0x29da06 thru 0x29f4ac (0x1aa7 bytes) 000162[1] 1763350434.975169 read 0x19daaa thru 0x19e4d9 (0xa30 bytes) 000164[0] 1763350435.003637 trunc from 0x5e12f5 to 0x82d682 (0x24c38e bytes) 000166[1] 1763350435.051414 read 0x4de099 thru 0x4dfa44 (0x19ac bytes) 000170[1] 1763350435.127328 read 0x19ba1b thru 0x19cec8 (0x14ae bytes) 000172[1] 1763350435.148287 write 0x1001d0 thru 0x1018db (0x170c bytes) 000174[0] 1763350435.186817 mapwrite 0x35784e thru 0x35894d (0x1100 bytes) 000175[1] 1763350435.248858 read 0x6c3080 thru 0x6c4478 (0x13f9 bytes) truncating to largest ever: 0x996190 000176[1] 1763350435.254361 trunc from 0x82d683 to 0x99618f (0x168b0d bytes) 000179[0] 1763350435.320368 read 0x7912df thru 0x7919ee (0x710 bytes) 000180[1] 1763350435.349202 mapread 0x7fb7cd thru 0x7fc052 (0x886 bytes) 000181[1] 1763350435.363433 trunc from 0x996190 to 0x006be1 (0x98f5ae bytes) 000182[1] 1763350435.410631 read 0x0056f8 thru 0x005e0b (0x714 bytes) 000188[1] 1763350435.489368 write 0x88306f thru 0x88466c (0x15fe bytes) 000190[1] 1763350435.520048 trunc from 0x88466d to 0x363db6 (0x5208b6 bytes) 000191[1] 1763350435.581464 write 0x512a27 thru 0x514765 (0x1d3f bytes) 000194[0] 1763350435.632081 mapread 0x102154 thru 0x1040b6 (0x1f63 bytes) 000196[0] 1763350435.777399 write 0x94e340 thru 0x94f338 (0xff9 bytes) 000201[0] 1763350435.943058 mapread 0x5313dd thru 0x532b4e (0x1772 bytes) 000202[0] 1763350436.003226 mapread 0x825d65 thru 0x825f14 (0x1b0 bytes) 000205[0] 1763350436.035583 mapwrite 0x57b1db thru 0x57b315 (0x13b bytes) 000209[0] 1763350436.137517 mapwrite 0x806cdd thru 0x807be9 (0xf0d bytes) 000210[0] 1763350436.214146 trunc from 0x94f339 to 0x0e9b9a (0x86579e bytes) 000215[1] 1763350436.294988 read 0x02c21b thru 0x02d1b8 (0xf9e bytes) 000217[0] 1763350436.363316 read 0x0a62f2 thru 0x0a6d17 (0xa26 bytes) 000219[0] 1763350436.406857 write 0x21a4e8 thru 0x21b8f0 (0x1409 bytes) 000224[0] 1763350436.501629 mapwrite 0x25164c thru 0x253163 (0x1b18 bytes) 000226[1] 1763350436.640567 mapread 0x05d589 thru 0x05e812 (0x128a bytes) 000227[1] 1763350436.736269 mapread 0x0fc176 thru 0x0fddb5 (0x1c40 bytes) 000231[0] 1763350436.799685 write 0x2d050d thru 0x2d1163 (0xc57 bytes) 000232[0] 1763350436.840077 mapread 0x26b6e2 thru 0x26cda1 (0x16c0 bytes) 000233[1] 1763350436.898610 mapwrite 0x57b576 thru 0x57c124 (0xbaf bytes) 000234[1] 1763350437.105635 trunc from 0x57c125 to 0x2cf96e (0x2ac7b6 bytes) 000236[0] 1763350437.135822 mapwrite 0x81010b thru 0x811aa2 (0x1998 bytes) 000237[1] 1763350437.321215 mapwrite 0x356c73 thru 0x358449 (0x17d7 bytes) 000238[1] 1763350437.455205 mapread 0x037a5e thru 0x03861a (0xbbd bytes) 000241[0] 1763350437.517161 mapread 0x516a55 thru 0x51814d (0x16f9 bytes) 000242[1] 1763350437.537891 mapwrite 0x8a354d thru 0x8a3d95 (0x849 bytes) 000243[0] 1763350437.662871 write 0x9a7132 thru 0x9a8b0a (0x19d9 bytes) 000244[1] 1763350437.699831 read 0x305259 thru 0x3065f0 (0x1398 bytes) 000246[1] 1763350437.733187 mapread 0x5c9c1e thru 0x5ca9b5 (0xd98 bytes) 000250[1] 1763350437.779361 read 0x26ffce thru 0x271a7e (0x1ab1 bytes) 000251[1] 1763350437.790579 mapread 0x3d19c6 thru 0x3d310f (0x174a bytes) 000252[0] 1763350437.835536 write 0x4c3122 thru 0x4c422c (0x110b bytes) 000254[1] 1763350437.907453 read 0x2bc692 thru 0x2bde48 (0x17b7 bytes) 000255[0] 1763350437.921399 write 0x168d29 thru 0x16ab11 (0x1de9 bytes) 000256[0] 1763350437.992614 write 0x38c42e thru 0x38d346 (0xf19 bytes) 000257[1] 1763350438.012448 read 0x77920f thru 0x77a588 (0x137a bytes) 000258[1] 1763350438.066209 read 0x84535c thru 0x846640 (0x12e5 bytes) 000259[0] 1763350438.113668 mapread 0x5a3c67 thru 0x5a5b97 (0x1f31 bytes) 000264[0] 1763350438.129714 write 0x422ea7 thru 0x4237db (0x935 bytes) 000267[0] 1763350438.186297 trunc from 0x9a8b0b to 0x7be95c (0x1ea1ae bytes) 000268[0] 1763350438.243723 read 0x180ef7 thru 0x181df7 (0xf01 bytes) 000269[0] 1763350438.277597 trunc from 0x7be95d to 0x401f87 (0x3bc9d5 bytes) 000272[0] 1763350438.309718 write 0x78a1f1 thru 0x78ac92 (0xaa2 bytes) 000274[1] 1763350438.335435 read 0x1b1213 thru 0x1b1ec3 (0xcb1 bytes) 000277[1] 1763350438.370056 mapwrite 0x81dc48 thru 0x81dfcc (0x385 bytes) 000279[0] 1763350438.529870 mapread 0x20a6b3 thru 0x20c1e1 (0x1b2f bytes) 000281[1] 1763350438.582335 mapwrite 0x1db25a thru 0x1dca85 (0x182c bytes) 000282[0] 1763350438.696402 read 0x0963bc thru 0x0970b5 (0xcfa bytes) 000285[0] 1763350438.733295 mapwrite 0x056b0b thru 0x0583aa (0x18a0 bytes) 000288[1] 1763350438.946793 mapwrite 0x202595 thru 0x203971 (0x13dd bytes) 000289[1] 1763350439.050339 write 0x7bf9d8 thru 0x7c0ea0 (0x14c9 bytes) 000290[0] 1763350439.077194 mapwrite 0x06624e thru 0x0674e4 (0x1297 bytes) 000292[1] 1763350439.179858 read 0x7cc891 thru 0x7cccf5 (0x465 bytes) 000296[1] 1763350439.257598 mapread 0x005033 thru 0x006560 (0x152e bytes) 000298[1] 1763350439.296643 trunc from 0x81dfcd to 0x771fb9 (0xac013 bytes) 000300[1] 1763350439.365260 write 0x25427f thru 0x255cd9 (0x1a5b bytes) 000301[1] 1763350439.374246 trunc from 0x771fba to 0x08cfff (0x6e4fba bytes) 000302[1] 1763350439.392158 write 0x006d1c thru 0x0079ae (0xc93 bytes) 000303[0] 1763350439.400990 trunc from 0x08d000 to 0x452d7a (0x3c5d7b bytes) 000304[0] 1763350439.446951 mapread 0x3017cc thru 0x30375c (0x1f91 bytes) 000305[0] 1763350439.469924 read 0x1b519a thru 0x1b6494 (0x12fb bytes) 000306[1] 1763350439.499178 mapread 0x0503ba thru 0x050fe7 (0xc2e bytes) 000307[0] 1763350439.537598 mapwrite 0x6e0902 thru 0x6e2497 (0x1b96 bytes) 000308[0] 1763350439.635307 mapread 0x357484 thru 0x357bac (0x729 bytes) 000313[0] 1763350439.805247 mapread 0x28b857 thru 0x28bb90 (0x33a bytes) 000314[1] 1763350439.825416 mapwrite 0x605bc0 thru 0x6078da (0x1d1b bytes) 000315[0] 1763350439.963929 mapread 0x541c09 thru 0x543806 (0x1bfe bytes) 000316[1] 1763350440.047207 trunc from 0x6e2498 to 0x94d09d (0x26ac06 bytes) 000317[1] 1763350440.098983 mapread 0x71a6fc thru 0x71ade0 (0x6e5 bytes) 000319[0] 1763350440.114214 write 0x3ece39 thru 0x3eed67 (0x1f2f bytes) 000321[0] 1763350440.198185 mapwrite 0x97bca1 thru 0x97d50e (0x186e bytes) 000322[0] 1763350440.340966 mapwrite 0x6f010b thru 0x6f04fa (0x3f0 bytes) 000325[0] 1763350440.485659 mapwrite 0x48a7bf thru 0x48c0ee (0x1930 bytes) 000327[0] 1763350440.646265 write 0x4cb350 thru 0x4cc56d (0x121e bytes) 000328[1] 1763350440.683303 trunc from 0x97d50f to 0x45b5d9 (0x521f35 bytes) 000329[1] 1763350440.759701 mapread 0x2a9755 thru 0x2aaebd (0x1769 bytes) 000330[1] 1763350440.799266 write 0x6975dd thru 0x69890f (0x1333 bytes) 000333[0] 1763350440.851488 mapwrite 0x42dc75 thru 0x42e188 (0x514 bytes) 000334[1] 1763350440.990977 mapwrite 0x91e44e thru 0x91e4ba (0x6d bytes) 000336[0] 1763350441.103872 trunc from 0x91e4bb to 0x02ecdf (0x8ef7db bytes) 000337[0] 1763350441.141585 mapwrite 0x705187 thru 0x7054ac (0x326 bytes) 000340[0] 1763350441.313698 read 0x60d453 thru 0x60e88b (0x1439 bytes) 000341[1] 1763350441.348542 mapread 0x1fa012 thru 0x1faa6e (0xa5d bytes) 000343[1] 1763350441.404320 read 0x154861 thru 0x1562a0 (0x1a40 bytes) 000345[1] 1763350441.435472 mapread 0x2cd489 thru 0x2cde5d (0x9d5 bytes) 000346[0] 1763350441.462362 write 0x56e6ad thru 0x56fb8b (0x14df bytes) 000347[0] 1763350441.499276 write 0x719d34 thru 0x71a090 (0x35d bytes) 000348[1] 1763350441.500261 write 0x3f743e thru 0x3f8b8c (0x174f bytes) 000349[1] 1763350441.541240 write 0x1463ca thru 0x14752d (0x1164 bytes) 000350[0] 1763350441.578714 trunc from 0x71a091 to 0x0b7bd1 (0x6624bf bytes) 000352[1] 1763350441.666320 mapwrite 0x203439 thru 0x204596 (0x115e bytes) 000354[0] 1763350441.809655 write 0x8174f3 thru 0x81818c (0xc9a bytes) 000355[1] 1763350441.875309 trunc from 0x81818d to 0x853b30 (0x3b9a4 bytes) 000357[1] 1763350441.931843 read 0x746586 thru 0x7484e6 (0x1f61 bytes) 000359[0] 1763350441.941863 trunc from 0x853b31 to 0x094fba (0x7beb76 bytes) 000360[1] 1763350441.988695 trunc from 0x094fbb to 0x585b40 (0x4f0b86 bytes) 000365[1] 1763350442.050236 mapwrite 0x96e0c4 thru 0x9700b1 (0x1fee bytes) 000366[0] 1763350442.171690 read 0x905f1a thru 0x907be6 (0x1ccd bytes) 000368[0] 1763350442.210378 write 0x39db69 thru 0x39ecf9 (0x1191 bytes) 000369[0] 1763350442.244248 mapread 0x76c64c thru 0x76e3af (0x1d64 bytes) 000370[1] 1763350442.298889 mapread 0x880906 thru 0x88212b (0x1826 bytes) 000379[1] 1763350442.355149 trunc from 0x9700b2 to 0x4bb910 (0x4b47a1 bytes) 000385[1] 1763350442.525978 mapwrite 0x120411 thru 0x120acd (0x6bd bytes) 000387[0] 1763350442.622505 mapwrite 0x5f57b7 thru 0x5f6ef6 (0x1740 bytes) 000388[1] 1763350442.746464 mapwrite 0x91a22e thru 0x91b282 (0x1055 bytes) 000390[1] 1763350443.000337 trunc from 0x91b283 to 0x947c0f (0x2c98d bytes) 000391[0] 1763350443.075346 mapwrite 0x474f16 thru 0x475802 (0x8ed bytes) 000392[0] 1763350443.179106 mapwrite 0x09594a thru 0x096654 (0xd0b bytes) 000393[0] 1763350443.273141 mapread 0x82fe85 thru 0x8300f9 (0x275 bytes) 000395[0] 1763350443.328277 write 0x536d74 thru 0x5388df (0x1b6c bytes) 000396[0] 1763350443.343624 write 0x9c8497 thru 0x9c9ca2 (0x180c bytes) 000397[0] 1763350443.378368 mapread 0x2a1c3c thru 0x2a29da (0xd9f bytes) 000398[1] 1763350443.440641 mapread 0x65c23f thru 0x65d34d (0x110f bytes) 000400[0] 1763350443.483326 read 0x4d5b61 thru 0x4d771f (0x1bbf bytes) 000403[0] 1763350443.516892 trunc from 0x9c9ca3 to 0x192bd7 (0x8370cb bytes) 000416[1] 1763350443.920452 mapwrite 0x3fa612 thru 0x3fb17b (0xb6a bytes) 000419[1] 1763350444.102320 write 0x5dd61f thru 0x5df13e (0x1b20 bytes) 000420[0] 1763350444.104074 trunc from 0x5df13f to 0x218192 (0x3c6fac bytes) 000422[0] 1763350444.200849 trunc from 0x218193 to 0x39a761 (0x1825cf bytes) 000423[1] 1763350444.230039 mapwrite 0x00b350 thru 0x00d2ea (0x1f9b bytes) 000424[1] 1763350444.322343 write 0x3568bb thru 0x3579fa (0x1140 bytes) 000431[1] 1763350444.478519 read 0x0ea858 thru 0x0ec563 (0x1d0c bytes) 000434[1] 1763350444.631416 mapwrite 0x851f47 thru 0x853992 (0x1a4c bytes) 000435[1] 1763350444.894191 mapread 0x0a6ac9 thru 0x0a7e93 (0x13cb bytes) 000436[0] 1763350444.916575 mapread 0x2eed26 thru 0x2f0c11 (0x1eec bytes) 000437[0] 1763350444.993625 read 0x30215c thru 0x303e8a (0x1d2f bytes) 000438[1] 1763350445.013941 write 0x09cbe6 thru 0x09e99d (0x1db8 bytes) truncating to largest ever: 0x9ff553 000439[1] 1763350445.035315 trunc from 0x853993 to 0x9ff552 (0x1abbc0 bytes) 000440[1] 1763350445.062722 mapread 0x52d5eb thru 0x52e253 (0xc69 bytes) 000441[1] 1763350445.100863 read 0x5e3d71 thru 0x5e5803 (0x1a93 bytes) 000442[1] 1763350445.102075 write 0x43ce99 thru 0x43e588 (0x16f0 bytes) 000443[0] 1763350445.103117 mapread 0x33b9c4 thru 0x33c9e7 (0x1024 bytes) 000444[0] 1763350445.146822 mapwrite 0x3d3e2b thru 0x3d53bc (0x1592 bytes) 000448[0] 1763350445.330499 mapread 0x65cdd4 thru 0x65eb98 (0x1dc5 bytes) 000452[0] 1763350445.410660 mapwrite 0x56c4d7 thru 0x56e436 (0x1f60 bytes) 000455[0] 1763350445.576822 trunc from 0x9ff553 to 0x5f46b9 (0x40ae99 bytes) 000456[0] 1763350445.622138 write 0x2bb528 thru 0x2bb8e5 (0x3be bytes) 000457[1] 1763350445.635637 trunc from 0x5f46ba to 0x0df94e (0x514d6b bytes) 000459[0] 1763350445.772590 write 0x7a71d1 thru 0x7a7215 (0x45 bytes) 000460[0] 1763350445.818852 mapwrite 0x309bcc thru 0x30b215 (0x164a bytes) 000462[0] 1763350445.929966 write 0x107e9d thru 0x108014 (0x178 bytes) 000463[1] 1763350445.972217 trunc from 0x7a7216 to 0x432720 (0x374af5 bytes) 000465[1] 1763350446.066729 read 0x10d907 thru 0x10f05a (0x1754 bytes) 000466[0] 1763350446.082992 trunc from 0x432721 to 0x587503 (0x154de3 bytes) 000469[0] 1763350446.171878 mapread 0x555820 thru 0x555ee7 (0x6c8 bytes) 000471[1] 1763350446.176259 trunc from 0x587504 to 0x529d82 (0x5d781 bytes) 000474[0] 1763350446.248052 trunc from 0x529d83 to 0x2aa17e (0x27fc04 bytes) 000475[0] 1763350446.295740 mapwrite 0x02025a thru 0x021722 (0x14c9 bytes) 000478[0] 1763350446.365866 trunc from 0x2aa17f to 0x1f0c98 (0xb94e6 bytes) 000479[0] 1763350446.386493 write 0x1335a5 thru 0x135438 (0x1e94 bytes) 000481[0] 1763350446.447075 read 0x19f7fc thru 0x1a0929 (0x112e bytes) 000482[0] 1763350446.496490 write 0x33b444 thru 0x33d30e (0x1ecb bytes) 000484[1] 1763350446.518112 read 0x2d7ac0 thru 0x2d8740 (0xc81 bytes) 000486[1] 1763350446.625618 trunc from 0x33d30f to 0x8fc676 (0x5bf368 bytes) 000487[1] 1763350446.696186 trunc from 0x8fc677 to 0x12120e (0x7db468 bytes) 000488[0] 1763350446.762050 mapwrite 0x305840 thru 0x3072ca (0x1a8b bytes) 000491[1] 1763350446.893567 mapwrite 0x8cbd0c thru 0x8cd37c (0x1671 bytes) 000493[1] 1763350447.129688 mapwrite 0x00b4fe thru 0x00c267 (0xd6a bytes) 000494[0] 1763350447.210471 trunc from 0x8cd37d to 0x89e7d5 (0x2eba7 bytes) 000495[1] 1763350447.286302 mapread 0x24b936 thru 0x24c270 (0x93b bytes) 000500[0] 1763350447.372152 write 0x695b54 thru 0x69742c (0x18d9 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 22:34:12 (1763350452) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 22:34:16 (1763350456) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.473716 s, 17.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.369283 s, 22.7 MB/s ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cd2050d8800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cd209739000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cd209739000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cd209739000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.10227 s, 11.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.62203 s, 40.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.5465 s, 41.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08378 s, 20.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.94234 s, 15.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.025 s, 34.7 MB/s PASS 16d (41s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 22:34:57 (1763350497) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.227315 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122322 s, 8.6 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 22:35:04 (1763350504) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 22:35:31 (1763350531) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 22:35:58 (1763350558) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 22:36:05 (1763350565) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 22:36:11 (1763350571) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.64704 s, 15.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.938003 s, 27.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.14513 s, 12.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.3799 s, 19.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98656 s, 13.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.59859 s, 16.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.01013 s, 13.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.951233 s, 27.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.12634 s, 23.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.34772 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53498 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.9999 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09407 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56201 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.58474 s, 10.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45682 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.81518 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27565 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.57522 s, 7.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (32s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 22:36:43 (1763350603) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! All operations completed A-OK! Sun Nov 16 22:37:18 EST 2025 + fsx 2 done: rc=0 Sun Nov 16 22:37:18 EST 2025 + fsx 3 done: rc=0 Sun Nov 16 22:37:18 EST 2025 + fsx 4 done: rc=0 Sun Nov 16 22:37:18 EST 2025 + fsx 5 done: rc=0 Sun Nov 16 22:37:18 EST 2025 PASS 16k (46s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 22:37:30 (1763350650) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 22:37:39 (1763350659) mmap test1: basic mmap operation (PASS, 0.16986s) mmap test2: MAP_PRIVATE not write back (PASS, 0.070749s) mmap test3: concurrent mmap ops on two nodes (PASS, 26.241s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4334s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1343s) mmap test6: check mmap write/read content on two nodes (PASS, 0.19387s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.90712s) mmap test11: multi-thread shared mmap (PASS, 0.92254s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0825s) PASS 18 (42s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:38:21 (1763350701) oleg254-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:38:25 (1763350705) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:38:31 (1763350711) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 22:38:37 (1763350717) now is 1763350719 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 new atime is 1763350781 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:39:45 (1763350785) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:39:53 (1763350793) PASS 24b (5s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:39:58 (1763350798) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:40:03 (1763350803) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 22:40:06 (1763350806) utime: good mknod times 1763350806 <= 1763350807 <= 1763350807 for /mnt/lustre/f26a utime: good mknod times 1763350806 <= 1763350807 <= 1763350807 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 22:40:12 (1763350812) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:40:20 (1763350820) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-11-16 22:40:22.000000000 -0500 Birth: 2025-11-16 22:40:23.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-11-16 22:40:22.000000000 -0500 Birth: 2025-11-16 22:40:23.000000000 -0500 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:40:28 (1763350828) 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:40:35 (1763350835) fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:40:41 (1763350841) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 22:41:00 (1763350860) -rw-r--r-- 1 root root 0 Nov 16 22:41 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:41:09 (1763350869) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42921 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:41:15 (1763350875) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:41:24 (1763350884) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:41:27 (1763350887) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:41:29 (1763350889) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:41:32 (1763350892) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 22:41:36 (1763350896) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid in FULL state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid in IDLE state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid in IDLE state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid in IDLE state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cd2050d8800.ost_server_uuid in IDLE state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cd2050d8800.ost_server_uuid in IDLE state after 0 sec PASS 34 (122s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:43:38 (1763351018) fail_loc=0x80000317 Race attempt 0 Wait for 48967 49048 for 60 sec... - open/close 832 (time 1763351033.10 total 10.00 last 83.19) - open/close 1635 (time 1763351043.11 total 20.01 last 80.22) - open/close 2468 (time 1763351053.11 total 30.02 last 83.25) - open/close 3254 (time 1763351063.13 total 40.03 last 78.49) total: 4000 open/close in 49.49 seconds: 80.83 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.154@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.154@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (69s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:44:47 (1763351087) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.961596 s, 54.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.767239 s, 68.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.814389 s, 64.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39086 s, 37.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4631 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7488512) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.828964 s, 63.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.954166 s, 54.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.987879 s, 53.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.991105 s, 52.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.994525 s, 52.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14728 s, 45.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7493632) after_dd(7444480) after(7495680) PASS 36 (283s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:49:30 (1763351370) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 - create 1555 (time 1763351383.28 total 10.02 last 155.13) - create 2981 (time 1763351393.28 total 20.03 last 142.52) - create 4516 (time 1763351403.28 total 30.03 last 153.50) - create 5872 (time 1763351413.29 total 40.03 last 135.53) - create 7432 (time 1763351423.29 total 50.04 last 155.93) - create 8928 (time 1763351433.30 total 60.04 last 149.56) total: 10000 create in 67.83 seconds: 147.43 ops/second PASS 37 (77s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:50:48 (1763351448) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:50:54 (1763351454) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:51:01 (1763351461) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 22:51:08 (1763351468) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:51:16 (1763351476) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (16s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:51:32 (1763351492) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:51:49 (1763351509) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:52:04 (1763351524) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (13s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:52:17 (1763351537) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:52:28 (1763351548) open: File exists Conflict PASS 41b (10s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:52:38 (1763351558) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (11s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:52:49 (1763351569) Conflict PASS 41d (10s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:52:59 (1763351579) Conflict PASS 41e (10s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:53:09 (1763351589) Conflict PASS 41f (10s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:53:20 (1763351600) Conflict PASS 41g (10s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:53:30 (1763351610) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (9s) == sanityn test 41i: reint_open: create vs create ======== 22:53:39 (1763351619) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (891s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 23:08:30 (1763352510) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (7s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 23:08:37 (1763352517) open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 23:08:44 (1763352524) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 23:08:51 (1763352531) Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 23:08:57 (1763352537) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 23:09:04 (1763352544) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 23:09:10 (1763352550) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 23:09:16 (1763352556) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 23:09:22 (1763352562) PASS 43a (32s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 23:09:54 (1763352594) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 23:10:01 (1763352601) Conflict PASS 43c (6s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 23:10:07 (1763352607) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 23:10:13 (1763352613) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 23:10:20 (1763352620) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (6s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 23:10:27 (1763352627) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 23:10:33 (1763352633) Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 23:10:39 (1763352639) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 23:10:41 (1763352641) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (54s) == sanityn test 43k: unlink vs create ==================== 23:11:35 (1763352695) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (529s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 23:20:24 (1763353224) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Nov 16 23:20:28 EST 2025 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 23:20:30 (1763353230) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 23:20:36 (1763353236) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 23:20:41 (1763353241) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 23:20:48 (1763353248) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 23:20:55 (1763353255) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 23:21:01 (1763353261) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 23:21:07 (1763353267) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 23:21:13 (1763353273) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 23:21:14 (1763353274) PASS 45a (48s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 23:22:02 (1763353322) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 23:22:09 (1763353329) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 23:22:16 (1763353336) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 23:22:21 (1763353341) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 23:22:27 (1763353347) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 23:22:34 (1763353354) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 23:22:40 (1763353360) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 23:22:45 (1763353365) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 23:22:46 (1763353366) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (468s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:30:35 (1763353835) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:30:40 (1763353840) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:30:45 (1763353845) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:30:51 (1763353851) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:30:56 (1763353856) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:31:01 (1763353861) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:31:06 (1763353866) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:31:11 (1763353871) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:31:17 (1763353877) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:31:18 (1763353878) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:31:19 (1763353879) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:31:20 (1763353880) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:31:21 (1763353881) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:31:22 (1763353882) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:31:23 (1763353883) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:31:24 (1763353884) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:31:25 (1763353885) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00561676 s, 1.8 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:31:32 (1763353892) 0+1 records in 0+1 records out 158 bytes copied, 0.00386382 s, 40.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (215938) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:31:36 (1763353896) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00333365 s, 307 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150744 s, 69.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:31:51 (1763353911) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00448061 s, 229 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:31:57 (1763353917) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159875 s, 65.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:32:01 (1763353921) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 23:32:05 (1763353925) llite.lustre-ffff9cd2050d8800.xattr_cache=1 llite.lustre-ffff9cd209739000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:06.000000000 -0500 Modify: 2025-11-16 23:32:06.000000000 -0500 Change: 2025-11-16 23:32:06.000000000 -0500 Birth: 2025-11-16 23:32:07.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:11.000000000 -0500 Modify: 2025-11-16 23:32:11.000000000 -0500 Change: 2025-11-16 23:32:11.000000000 -0500 Birth: 2025-11-16 23:32:12.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:16.000000000 -0500 Modify: 2025-11-16 23:32:16.000000000 -0500 Change: 2025-11-16 23:32:16.000000000 -0500 Birth: 2025-11-16 23:32:18.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:16.000000000 -0500 Modify: 2025-11-16 23:32:17.000000000 -0500 Change: 2025-11-16 23:32:17.000000000 -0500 Birth: 2025-11-16 23:32:17.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:22.000000000 -0500 Modify: 2025-11-16 23:32:22.000000000 -0500 Change: 2025-11-16 23:32:22.000000000 -0500 Birth: 2025-11-16 23:32:23.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:32:22.000000000 -0500 Modify: 2025-11-16 23:32:22.000000000 -0500 Change: 2025-11-16 23:32:22.000000000 -0500 Birth: 2025-11-16 23:32:23.000000000 -0500 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 23:32:29 (1763353949) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 23:32:37 (1763353957) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 23:32:44 (1763353964) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 23:32:57 (1763353977) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:33:07 (1763353987) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:33:08 (1763353988) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 23:33:21 (1763354001) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:33:35 (1763354015) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1763354015 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.65387 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (25s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:34:00 (1763354040) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1763354042 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.59042 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (39s) == sanityn test 60: Verify data_version behaviour ======== 23:34:39 (1763354079) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 23:34:41 (1763354081) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:34:44 (1763354084) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:34:48 (1763354088) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00332602 s, 19.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:34:49 (1763354089) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00409269 s, 16.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:34:50 (1763354090) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:34:51 (1763354091) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0115272 s, 5.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:34:52 (1763354092) llite.lustre-ffff9cd2050d8800.xattr_cache=1 llite.lustre-ffff9cd209739000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:34:55 (1763354095) llite.lustre-ffff9cd2050d8800.xattr_cache=1 llite.lustre-ffff9cd209739000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:34:57 (1763354097) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00300596 s, 341 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00465462 s, 220 kB/s 228695: get lock2 228696: get lock1 228696: done 228696: get lock2 228695: done 228695: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 228695: cannot get lock1: Resource deadlock avoided 228696: done 228696: exit rc=0 228695: done 228695: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:35:03 (1763354103) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.023302 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110726 s, 94.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229276 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 23:35:10 (1763354110) opening files: ........ closing files: ........ PASS 76 (19s) == sanityn test 77a: check FIFO NRS policy =============== 23:35:29 (1763354129) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141787 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221891 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356316 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426202 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476184 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752067 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864086 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597039 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597659 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102999 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921733 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104579 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989466 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991438 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888797 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106017 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105209 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216002 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861629 s, 122 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114949 s, 91.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394049 s, 266 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258442 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295217 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209172 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317636 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252829 s, 41.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262193 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293438 s, 35.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233935 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394871 s, 266 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201985 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00873732 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134461 s, 78.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 23:35:32 (1763354132) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150975 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248885 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541521 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057102 s, 18.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719473 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472681 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767626 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704612 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712172 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806971 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101507 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102262 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135689 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205799 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119347 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125987 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136038 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12979 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307976 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194647 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190827 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146537 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259797 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205033 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186546 s, 56.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911073 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262127 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294004 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244837 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176749 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166304 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01468 s, 71.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018435 s, 56.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132184 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325409 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036132 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346833 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050392 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742544 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742634 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640115 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481603 s, 21.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512449 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594938 s, 17.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828442 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071687 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901867 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936973 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997753 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117597 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147684 s, 71.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149513 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136572 s, 76.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206439 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022708 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177889 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016975 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198819 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244592 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033015 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177417 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891128 s, 118 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027624 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182181 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138779 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039462 s, 266 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 23:35:36 (1763354136) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17033 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439585 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412274 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885242 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898044 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914249 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984449 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120623 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121823 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101727 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137002 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107886 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116553 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110202 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142768 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129476 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126247 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110963 s, 94.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017426 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177566 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015886 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242611 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151618 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018 s, 58.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267686 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014664 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266727 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407641 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409276 s, 256 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149646 s, 70.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130107 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284914 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266089 s, 394 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.208494 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340658 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381144 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977429 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1734 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157982 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17565 s, 6.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180796 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168255 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177966 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178283 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180282 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166229 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181837 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176226 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177462 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199819 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025659 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029023 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194206 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023292 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355404 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0076214 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357019 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218665 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031615 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239541 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053161 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284415 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027172 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350064 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411847 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00616546 s, 170 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134647 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411973 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557272 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625824 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917978 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111025 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110798 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101152 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114219 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123214 s, 8.5 MB/sldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116568 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121654 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999326 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117584 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111945 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140553 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138842 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139935 s, 74.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166045 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274222 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175761 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176291 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177846 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190311 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195403 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210054 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223486 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214036 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021987 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228375 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276076 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265731 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267161 s, 39.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 23:35:41 (1763354141) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168171 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326913 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422934 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527362 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568642 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605422 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494717 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479387 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067828 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573996 s, 18.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935194 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866717 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915528 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101463 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112937 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119055 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106142 s, 98.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137491 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301578 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121613 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187759 s, 55.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297597 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284063 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217217 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033359 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150982 s, 69.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251963 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671826 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991141 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216861 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147228 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364928 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897115 s, 117 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125308 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238407 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022881 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816044 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887103 s, 11.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923387 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921154 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930683 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868464 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103155 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845011 s, 12.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937923 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906367 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107679 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105583 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105397 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132599 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175642 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202947 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189523 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211873 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179761 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179522 s, 58.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156299 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145651 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378404 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985189 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236707 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982253 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183601 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594671 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046868 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500261 s, 210 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123088 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037498 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463941 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03836 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365364 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484526 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434506 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752228 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498698 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572335 s, 18.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091247 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890818 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819517 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770355 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930275 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724535 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101344 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169248 s, 62.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158729 s, 66.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112732 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015828 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170683 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135809 s, 77.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167871 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205463 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021354 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244974 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264877 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363015 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00600261 s, 175 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262283 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862853 s, 122 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672451 s, 156 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 23:35:46 (1763354146) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.54@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118229 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230945 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216765 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03912 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323473 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849912 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811502 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845172 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147175 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225356 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073248 s, 143 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104591 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116609 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121031 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164382 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235796 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169453 s, 61.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132054 s, 79.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148376 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015038 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00345255 s, 304 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105383 s, 99.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176838 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123281 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369305 s, 284 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351674 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120727 s, 86.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138567 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14959 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151664 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157082 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161164 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170709 s, 6.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13089 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347041 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684467 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516949 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675511 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744049 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587421 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767511 s, 13.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788682 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098956 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091811 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961233 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094662 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852549 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994573 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108991 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11357 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275456 s, 3.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268448 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268425 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266368 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273438 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275969 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28125 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274702 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282249 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282093 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277266 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279154 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289027 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278427 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278185 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299105 s, 3.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14806 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026146 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384432 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459655 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049428 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357504 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606361 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587804 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716415 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088665 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1025 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888098 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093625 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875531 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958048 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888534 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907521 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145405 s, 72.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109897 s, 95.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748482 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259958 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200287 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129937 s, 80.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242369 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153906 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135974 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949454 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698766 s, 150 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197864 s, 53.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975288 s, 108 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141033 s, 74.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560133 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755338 s, 139 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126847 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244511 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729767 s, 14.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770684 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742917 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707983 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864332 s, 12.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842454 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884913 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821308 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980751 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560291 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887237 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857681 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101952 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940384 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104517 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245856 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208376 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279185 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020356 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296592 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301782 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274913 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212837 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296699 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230943 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028071 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259837 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256283 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337732 s, 31.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309691 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337442 s, 31.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=4 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 23:35:54 (1763354154) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181693 s, 92.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265118 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418179 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894967 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094527 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779009 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953198 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108937 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096764 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101155 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975053 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101841 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975345 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978634 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917452 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116347 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111341 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989575 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102324 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921939 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124196 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126248 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212831 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144837 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988732 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022206 s, 47.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217667 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577071 s, 182 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221561 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224396 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215275 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246782 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005542 s, 189 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139636 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384485 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635151 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748917 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106378 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116622 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141669 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126768 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121485 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118714 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122446 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128894 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113951 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126521 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138061 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117171 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142437 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496813 s, 211 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216781 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023847 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020696 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132075 s, 79.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895049 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163242 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011246 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237102 s, 44.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237161 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036575 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180473 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028794 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756454 s, 139 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306464 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594634 s, 176 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151299 s, 111 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323116 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364935 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305231 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676216 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646332 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072216 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101746 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918641 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093118 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939465 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100897 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808809 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074184 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782047 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842108 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849863 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143933 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135139 s, 77.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154811 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129368 s, 81.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135836 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191288 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141686 s, 74.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135325 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108444 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353783 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884398 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276905 s, 379 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220293 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00333743 s, 314 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0083027 s, 126 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318374 s, 32.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133665 s, 126 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274696 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052392 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505721 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829523 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694233 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679392 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959737 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088306 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108853 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100925 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884518 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114371 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109882 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964336 s, 10.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10436 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102071 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011216 s, 93.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112234 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219548 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294373 s, 356 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272503 s, 38.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cd2050d8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cd2050d8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00309576 s, 339 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026484 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227511 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410144 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327753 s, 320 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348792 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057263 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332558 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124053 s, 84.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147417 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135714 s, 77.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 23:36:01 (1763354161) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 23:36:05 (1763354165) oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg254-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg254-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg254-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg254-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 23:36:09 (1763354169) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 23:36:16 (1763354176) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.102992 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219237 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313411 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402355 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551093 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516114 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055564 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466109 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773537 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912092 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957594 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852604 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110816 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094353 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107757 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967521 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103289 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360041 s, 29.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324864 s, 32.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412644 s, 25.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400335 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380131 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330852 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332368 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410462 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039356 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385303 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403044 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471934 s, 22.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420824 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406908 s, 25.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439416 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454372 s, 23.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9624 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4308 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 23:37:02 (1763354222) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22377 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729274 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680799 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740536 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861265 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072528 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698444 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739206 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612068 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159141 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159109 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149398 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16824 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177517 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144959 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200318 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173046 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251764 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248778 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145626 s, 72.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100464 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169422 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377409 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368007 s, 285 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00835132 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362662 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104417 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309292 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160096 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400824 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02518 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424494 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477202 s, 220 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8281 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9684 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22402 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830802 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831069 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705827 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730622 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868183 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102825 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743526 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878726 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153662 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153607 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142051 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157202 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165088 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161423 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156037 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182333 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137435 s, 76.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019787 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020396 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195487 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222122 s, 47.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284488 s, 36.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198625 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237882 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010148 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335085 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188629 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241873 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254787 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249694 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440681 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360832 s, 291 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8565 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9509 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 23:39:15 (1763354355) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22554 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064419 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548208 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635363 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829534 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746644 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06424 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840314 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750847 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159001 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158936 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141807 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17483 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159723 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165596 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18012 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163331 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156844 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174518 s, 60.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019419 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142915 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178631 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138955 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155794 s, 67.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227071 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153809 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276514 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00595238 s, 176 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498766 s, 210 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284889 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664123 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114437 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232165 s, 452 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8358 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9528 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22172 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043725 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394703 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759639 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931695 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873264 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101251 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797444 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770549 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159582 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15952 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149843 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15886 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16389 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134976 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165076 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189062 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191449 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180996 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184006 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020881 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226668 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206481 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128267 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228129 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254008 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533744 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211189 s, 49.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278451 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255054 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283336 s, 37.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427299 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00691391 s, 152 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8436 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9562 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 23:41:28 (1763354488) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.263588 s, 63.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502841 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531912 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460639 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774411 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071981 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762565 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804158 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112325 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101142 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772218 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107764 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100367 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942289 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895686 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105133 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110902 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021097 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201551 s, 52.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224372 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214757 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325781 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312255 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132887 s, 78.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03438 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279615 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367699 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154506 s, 67.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195106 s, 53.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125242 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155609 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393194 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228663 s, 45.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98699 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73263 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.54@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.54@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244832 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318923 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806035 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738843 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820337 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889173 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106446 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120369 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143469 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143781 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157202 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166989 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156816 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257257 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242798 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275911 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269396 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014943 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210406 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202965 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206906 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196793 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243349 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171253 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916497 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288093 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00559382 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579642 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887708 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326084 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287443 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208793 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00564775 s, 186 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96086 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73454 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.54@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241028 s, 69.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383309 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638397 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774066 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820182 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725075 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681652 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760172 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881638 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787587 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721173 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976955 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825369 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129673 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113618 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105469 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133619 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122732 s, 85.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164954 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139899 s, 75.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206435 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177863 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235442 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130868 s, 80.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270593 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381555 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759704 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246833 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222174 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130017 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158871 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00273063 s, 384 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243125 s, 431 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01605 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88321 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444167 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049174 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409475 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556992 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614736 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046761 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683759 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505622 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594035 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449353 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895592 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851971 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948789 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339764 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369195 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401818 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420379 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134576 s, 77.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174509 s, 60.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167193 s, 62.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202826 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223101 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401457 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024687 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275564 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293437 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287041 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047879 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262085 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282579 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301224 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467422 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302654 s, 34.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76939 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73205 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92344 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73739 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.837633 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051658 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578843 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456615 s, 23.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575187 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393216 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421463 s, 24.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525289 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484026 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157237 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147918 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178894 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185197 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189848 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198757 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204264 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206731 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160408 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018354 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178862 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151726 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156892 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161508 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021605 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418046 s, 251 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250296 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528883 s, 198 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243373 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245069 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110447 s, 94.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719097 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00938711 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029781 s, 35.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6944 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4357 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241267 s, 69.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288297 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398619 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388593 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859574 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112118 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119722 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102922 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12747 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158827 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150512 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138991 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159566 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144278 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132536 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154854 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160166 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017785 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173814 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168076 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016844 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146617 s, 71.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151379 s, 69.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187141 s, 56.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020436 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826679 s, 127 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215332 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571175 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250348 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264334 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00863412 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663264 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00218804 s, 479 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99572 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73118 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449784 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305296 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638379 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767722 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690028 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10512 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984087 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968724 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11505 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107871 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134307 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113822 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122192 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147947 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145084 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348965 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366287 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175731 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159575 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177319 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177086 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229283 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023293 s, 45.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243128 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253253 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307266 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222798 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162566 s, 64.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00727186 s, 144 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107636 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897478 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140693 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720099 s, 146 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg254-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg254-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7933 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73217 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77419 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73112 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (363s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 23:47:31 (1763354851) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256282 s, 65.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222196 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316545 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840552 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082309 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11241 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982371 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105097 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881635 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971641 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100602 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114049 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123686 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111036 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119822 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117004 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11998 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110982 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01493 s, 70.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146597 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137552 s, 76.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118453 s, 88.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020384 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233623 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209196 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829658 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245542 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826128 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00995827 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00576753 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144024 s, 72.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109849 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00854063 s, 123 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02599 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73152 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (38s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 23:48:09 (1763354889) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.54@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.54@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.250383 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615063 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876648 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113924 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955002 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117624 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120782 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138121 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151336 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143239 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127097 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150034 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142607 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235912 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243223 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300876 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274388 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189201 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0095508 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464207 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194665 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189774 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204058 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191921 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354406 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181948 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189409 s, 55.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261769 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220503 s, 47.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676273 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405383 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434706 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689925 s, 152 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1516 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73065 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 23:48:50 (1763354930) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.54@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244735 s, 68.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433285 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510746 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604496 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737677 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767404 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07975 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730678 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109101 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761079 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911007 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978756 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093338 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12236 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105496 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114457 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100875 s, 10.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016303 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185136 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171756 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216812 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018121 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159079 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414436 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020802 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110687 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044494 s, 236 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183873 s, 57.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249282 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022005 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453669 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358133 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312417 s, 336 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96932 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (31s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 23:49:21 (1763354961) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44759 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240574 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507201 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616767 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702648 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516601 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06461 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640081 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894747 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10459 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669529 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103956 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094469 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271827 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243468 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315623 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310099 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151364 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017098 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161888 s, 64.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180029 s, 58.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188719 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107189 s, 97.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00819964 s, 128 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189675 s, 55.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241661 s, 43.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241667 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278463 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00562774 s, 186 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098323 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101522 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823706 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085636 s, 122 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89377 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73282 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92057 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73051 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:50:42 (1763355042) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.86473 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439412 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412603 s, 25.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611505 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664536 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776493 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601883 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795113 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708231 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1427 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181863 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185452 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181872 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197182 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207705 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215028 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198654 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196248 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197487 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236071 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305441 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215536 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301967 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174208 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339666 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011247 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031906 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909927 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273364 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204279 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179383 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135284 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111375 s, 94.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7275 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9454 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:51:48 (1763355108) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241296 s, 69.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031911 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301927 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100209 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879974 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960796 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085195 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095218 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094465 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106421 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105949 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981853 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104596 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129236 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115522 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127161 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111546 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157214 s, 66.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222316 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024389 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198508 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125489 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187954 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253942 s, 41.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031118 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112643 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138205 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196923 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00880987 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200494 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142391 s, 73.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746957 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487222 s, 215 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06259 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73256 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg254-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.442575 s, 37.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271803 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586006 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422083 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382317 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568044 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953364 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900486 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891662 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105098 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999991 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119754 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102062 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196357 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306955 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304467 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110675 s, 94.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139659 s, 75.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212273 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197954 s, 53.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269107 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117442 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027816 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101132 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295121 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00847816 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031746 s, 33.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155914 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123763 s, 84.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315818 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185474 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103681 s, 101 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8028 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73505 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78693 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73095 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (119s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 23:53:47 (1763355227) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre opts: Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg254-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg254-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.54,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg254-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.54,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/57295/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.54,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg254-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cd20a145000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cd20a2d1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd20a145000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd20a2d1000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.243646 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256023 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500255 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826801 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780801 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731809 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975663 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894122 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584099 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848994 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837937 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130218 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146763 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240252 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248997 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292226 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313847 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478714 s, 219 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136989 s, 76.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176041 s, 59.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208368 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167975 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160707 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194593 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234743 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279387 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506775 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233744 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130439 s, 80.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638697 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705618 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059692 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444477 s, 236 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99805 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96991 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256569 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265215 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374585 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671128 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696565 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572101 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721877 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935397 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945674 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103872 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967849 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105756 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10036 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10801 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108916 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257454 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242078 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141094 s, 74.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196085 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179759 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153129 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178526 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199612 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017736 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397093 s, 264 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173314 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388678 s, 270 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768698 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142803 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386184 s, 272 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101529 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637526 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00188764 s, 555 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03291 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7319 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (113s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 23:55:40 (1763355340) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg254-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 23:55:48 (1763355348) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 23:55:51 (1763355351) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg254-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.07148 s, 341 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg254-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.027 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 23:56:42 (1763355402) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133218 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315478 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518137 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093213 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10281 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916882 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833392 s, 12.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918347 s, 11.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cd20a145000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883884 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100197 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105724 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926875 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120919 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113802 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105922 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113265 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215012 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163005 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333347 s, 31.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9cd20a145000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cd20a145000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312328 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00701509 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036628 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289698 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405895 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183358 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377052 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392595 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365783 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555647 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101741 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531976 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566834 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276985 s, 379 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96358 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97096 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251573 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711571 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837503 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120982 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993293 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128927 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14791 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140464 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139205 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167811 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151413 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169226 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158134 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212272 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229803 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297217 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293947 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154086 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169055 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169166 s, 62.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288715 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019766 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219711 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350581 s, 299 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270481 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190146 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103412 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114572 s, 91.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804548 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622864 s, 168 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0056887 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014803 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00187312 s, 560 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg254-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg254-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99823 s, 21.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97157 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 23:57:48 (1763355468) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 23:57:52 (1763355472) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (39s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 23:58:32 (1763355512) mds.MDS.mdt.nrs_policies=tbf oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg254-client: oleg254-server: ssh exited with exit code 36 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 oleg254-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg254-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg254-client: oleg254-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 23:58:44 (1763355524) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 270033 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:59:26 (1763355566) ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 23:59:29 (1763355569) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935085 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 23:59:29.000000000 -0500 Modify: 2025-11-16 23:59:29.000000000 -0500 Change: 2025-11-16 23:59:29.000000000 -0500 Birth: 2025-11-16 23:59:30.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:59:42 (1763355582) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:59:43 (1763355583) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:59:44 (1763355584) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:59:45 (1763355585) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:59:46 (1763355586) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:59:47 (1763355587) total: 2500 open/close in 5.53 seconds: 451.72 ops/second snapshot_time 1763355612.751816135 secs.nsecs start_time 1763355589.614297227 secs.nsecs elapsed_time 23.137518908 secs.nsecs open 2505 samples [usecs] 86 2525 801702 292594352 close 2504 samples [usecs] 66 920 257285 31300501 mknod 2501 samples [usecs] 174 2414 698555 225998191 unlink 2500 samples [usecs] 280 11114 1128949 837437251 mkdir 2 samples [usecs] 1219 1627 2846 4133090 rmdir 2 samples [usecs] 1965 3878 5843 18900109 rename 2500 samples [usecs] 259 34604 3217570 6879777720 getattr 3760 samples [usecs] 7 637 85081 2735735 parallel_rename_file 2500 samples [usecs] 259 34604 3217570 6879777720 crossdir_rename 2500 samples [usecs] 259 34604 3217570 6879777720 PASS 81d (26s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:00:13 (1763355613) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7582 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7582 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:00:15 (1763355615) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:00:16 (1763355616) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 00:00:24 (1763355624) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1763355624 Finishing test test1 at 1763355625 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:00:26 (1763355626) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 00:00:28 (1763355628) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:00:29 (1763355629) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:00:30 (1763355630) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5755 0x167b 0x240000400 1 5753 0x1679 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5756 0x167c 0x240000400 1 5754 0x167a 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 00:00:39 (1763355639) osc.lustre-OST0000-osc-ffff9cd20a145000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9cd20a2d1000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cd20a145000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cd20a2d1000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0426296 s, 192 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.0941 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 287561 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0159306 s, 257 kB/s osc.lustre-OST0000-osc-ffff9cd20a145000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cd20a2d1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd20a145000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd20a2d1000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:00:49 (1763355649) llite.lustre-ffff9cd20a145000.fast_read=0 llite.lustre-ffff9cd20a2d1000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0226611 s, 92.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9cd20a145000.fast_read=1 llite.lustre-ffff9cd20a2d1000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:01:04 (1763355664) llite.lustre-ffff9cd20a145000.fast_read=0 llite.lustre-ffff9cd20a2d1000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00157303 s, 7.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9cd20a145000.fast_read=1 llite.lustre-ffff9cd20a2d1000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:01:12 (1763355672) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:01:13 (1763355673) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00138892 s, 2.9 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:01:15 (1763355675) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0483616 s, 43.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:01:18 (1763355678) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.052616 s, 39.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 00:01:20 (1763355680) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00175566 s, 6.8 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 00:01:22 (1763355682) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000607434 s, 6.7 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 00:01:25 (1763355685) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000488632 s, 8.4 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 00:01:27 (1763355687) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000525072 s, 7.8 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 00:01:31 (1763355691) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1763355692 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1763355694 Finishing test test23 at 1763355694 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 00:01:38 (1763355698) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 00:01:39 (1763355699) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 00:01:57 (1763355717) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:01:58 (1763355718) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00389056 s, 263 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937620 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-17 00:01:59.000000000 -0500 Modify: 2025-11-17 00:01:59.000000000 -0500 Change: 2025-11-17 00:01:59.000000000 -0500 Birth: 2025-11-17 00:02:00.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937620 -rw-r--r-- 1 0 0 1763355720 1763355719 1763355719 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937620 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-17 00:01:59.000000000 -0500 Modify: 2025-11-17 00:01:59.000000000 -0500 Change: 2025-11-17 00:01:59.000000000 -0500 Birth: 2025-11-17 00:02:00.000000000 -0500 1763355719 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 00:02:01 (1763355721) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 00:02:03 (1763355723) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0986617 s, 106 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5760 0x1680 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:02:07 (1763355727) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0954573 s, 110 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5765 0x1685 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7582 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 00:02:16 (1763355736) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00207171 s, 4.8 MB/s llite.lustre-ffff9cd20a145000.hybrid_io=0 llite.lustre-ffff9cd20a2d1000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.28137 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:02:22 (1763355742) 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre 192.168.202.154@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 PASS 109 (499s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:10:41 (1763356241) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-17 00:10:42.000000000 -0500 Modify: 2025-11-17 00:10:42.000000000 -0500 Change: 2025-11-17 00:10:42.000000000 -0500 Birth: 2025-11-17 00:10:43.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg254-client.virtnet oleg254-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:11:46 (1763356306) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 00:11:48 (1763356308) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 00:11:49 (1763356309) lustre-OST0000-osc-ffff9cd2107a5800 active. lustre-OST0001-osc-ffff9cd2107a5800 active. lustre-OST0000-osc-ffff9cd2107a5800 active. lustre-OST0001-osc-ffff9cd2107a5800 active. MGC192.168.202.154@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 00:11:51 (1763356311) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:11:52 (1763356312) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:11:53 (1763356313) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:11:54 (1763356314) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0894 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.102 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0977 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1052 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1008 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1043 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1022 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1454 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1506 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.153 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.169 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1558 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1754 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1717 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2401 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2617 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2423 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2558 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cd20a142000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cd20a142000.ost_server_uuid in IDLE state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 6437 sec ============== 00:13:01 (1763356381) === sanityn: start cleanup 00:13:01 (1763356381) === Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg254-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:13:46 (1763356426) ===