-----============= acceptance-small: sanityn ============----- Mon Dec 22 04:01:24 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg439-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 04:01:32 (1766394092) === Starting client oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Mount client oleg439-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Started clients oleg439-client.virtnet: 192.168.204.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg439-client.virtnet: executing check_config_client /mnt/lustre oleg439-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg439-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93f847fb8000.idle_timeout=debug osc.lustre-OST0000-osc-ffff93f8483b5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f847fb8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f8483b5000.idle_timeout=debug disable quota as required oleg439-server: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 04:01:48 (1766394108) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143635 s, 73.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7565] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 04:01:49 (1766394109) PASS 0 (4s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 04:01:53 (1766394113) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 04:01:56 (1766394116) -rw-r--r-- 1 root root 0 Dec 22 04:01 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 04:01:58 (1766394118) -rw-r--r-- 1 root root 0 Dec 22 04:01 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 04:02:01 (1766394121) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 04:02:04 (1766394124) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 04:02:06 (1766394126) total 1 -rwxrwxrwx 1 root root 0 Dec 22 04:01 f2a -rwxrwxrwx 1 root root 0 Dec 22 04:01 f2b total 1 -rwxrwxrwx 1 root root 0 Dec 22 04:01 f2a -rwxrwxrwx 1 root root 0 Dec 22 04:01 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00624819 s, 81.9 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 04:02:09 (1766394129) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 04:02:10 (1766394130) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129136 s, 16.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 04:02:13 (1766394133) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 04:02:15 (1766394135) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 04:02:19 (1766394139) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 04:02:21 (1766394141) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 04:02:24 (1766394144) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 04:02:26 (1766394146) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 04:02:28 (1766394148) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 04:02:30 (1766394150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164082 s, 63.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.00909197 s, 0.3 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 04:02:32 (1766394152) 1+0 records in 1+0 records out 1 byte copied, 0.00140485 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00286917 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00319615 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00275514 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00250287 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00218838 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00239966 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00255698 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00241131 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00234846 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00252152 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00280146 s, 0.4 kB/s PASS 10a (2s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 04:02:34 (1766394154) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00262706 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00852159 s, 481 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0002566 s, 12.0 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 04:02:36 (1766394156) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 04:02:39 (1766394159) warning: '-runtime' deprecated, use '-t runtime' instead - link 1800 (time 1766394169.88 total 10.00 last 180.00) - link 3427 (time 1766394179.88 total 20.00 last 162.68) - link 5694 (time 1766394189.88 total 30.01 last 226.60) - link 8079 (time 1766394199.89 total 40.01 last 238.44) - link 10000 (time 1766394208.05 total 48.17 last 235.29) - link 12536 (time 1766394218.05 total 58.17 last 253.55) - link 15016 (time 1766394228.06 total 68.18 last 247.91) - link 17330 (time 1766394238.06 total 78.18 last 231.38) - link 19800 (time 1766394248.06 total 88.18 last 246.93) - link 22085 (time 1766394258.06 total 98.19 last 228.39) total: 22429 link in 100.00 seconds: 224.28 ops/second using seed 1355688032 running for 100 seconds - stat 10000 (time 1766394169 ; total 9 ; last 9) - stat 20000 (time 1766394177 ; total 17 ; last 8) - stat 30000 (time 1766394184 ; total 24 ; last 7) - stat 40000 (time 1766394191 ; total 31 ; last 7) - stat 50000 (time 1766394198 ; total 38 ; last 7) - stat 60000 (time 1766394204 ; total 44 ; last 6) - stat 70000 (time 1766394212 ; total 52 ; last 8) - stat 76169 (time 1766394223 ; total 63 ; last 11) - stat 80000 (time 1766394229 ; total 69 ; last 6) - stat 86013 (time 1766394240 ; total 80 ; last 11) - stat 90000 (time 1766394246 ; total 86 ; last 6) - stat 95848 (time 1766394257 ; total 97 ; last 11) total: 97675 stats in 100 seconds: 976.750000 stats/second - unlinked 0 (time 1766394210 ; total 0 ; last 0) - unlinked 10000 (time 1766394266 ; total 56 ; last 56) - unlinked 20000 (time 1766394293 ; total 83 ; last 27) unlink(/mnt/lustre2/lockdir/lockfile22429) error: No such file or directory total: 22428 unlinks in 89 seconds: 252.000000 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21341) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21343) - No such process PASS 12 (141s) == sanityn test 13: test directory page revocation ======= 04:05:00 (1766394300) f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 04:05:02 (1766394302) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22528 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 04:05:04 (1766394304) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23108 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 04:05:06 (1766394306) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23686 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 04:05:08 (1766394308) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24267 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 04:05:10 (1766394310) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24848 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 04:05:13 (1766394313) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7524352 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7524352KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (28s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 04:05:41 (1766394341) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0855093 s, 98.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0637614 s, 132 MB/s Chance of close/open is 1 in 50 Seed set to 4342 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[1] 1766394342.891435 mapwrite 0x8315f0 thru 0x838fe5 (0x79f6 bytes) truncating to largest ever: 0x74bf8b 000003[0] 1766394342.919666 trunc from 0x838fe6 to 0x74bf8a (0xed05b bytes) 000005[0] 1766394342.936593 mapwrite 0x427afb thru 0x429027 (0x152d bytes) 000006[1] 1766394342.954665 read 0x1b2d9c thru 0x1b8a65 (0x5cca bytes) 000007[0] 1766394342.963511 mapwrite 0x238410 thru 0x23cb81 (0x4772 bytes) truncating to largest ever: 0x9f91f1 000008[0] 1766394342.988509 trunc from 0x74bf8b to 0x9f91f0 (0x2ad266 bytes) 000009[0] 1766394342.994813 write 0x11a1eb thru 0x120dd4 (0x6bea bytes) 000010[0] 1766394342.998013 write 0x892008 thru 0x898257 (0x6250 bytes) 000011[0] 1766394343.006735 write 0x2ae545 thru 0x2bba05 (0xd4c1 bytes) 000013[1] 1766394343.010258 mapread 0x4eeee1 thru 0x4fa382 (0xb4a2 bytes) 000017[1] 1766394343.036030 mapwrite 0x24674b thru 0x2559df (0xf295 bytes) 000018[1] 1766394343.064004 read 0x8af5d7 thru 0x8b3ebf (0x48e9 bytes) 000019[0] 1766394343.066629 write 0x1d78a7 thru 0x1e247d (0xabd7 bytes) 000020[0] 1766394343.076083 mapread 0x58f0a5 thru 0x59d9e6 (0xe942 bytes) 000023[0] 1766394343.096714 read 0x6af8a0 thru 0x6b0bbb (0x131c bytes) 000024[1] 1766394343.101463 mapwrite 0x48ee5f thru 0x493a95 (0x4c37 bytes) 000025[0] 1766394343.125790 mapread 0x0d4850 thru 0x0e20e0 (0xd891 bytes) 000027[1] 1766394343.140978 mapread 0x60de6a thru 0x610425 (0x25bc bytes) 000029[1] 1766394343.153046 trunc from 0x9f91f1 to 0x7c2e23 (0x2363cd bytes) 000030[0] 1766394343.165716 write 0x100566 thru 0x1037ce (0x3269 bytes) 000031[0] 1766394343.173470 read 0x4956e6 thru 0x49ac18 (0x5533 bytes) 000033[0] 1766394343.179935 mapread 0x470dc6 thru 0x472519 (0x1754 bytes) 000034[0] 1766394343.190278 mapwrite 0x98a8cf thru 0x98c75f (0x1e91 bytes) 000037[0] 1766394343.219293 trunc from 0x98c760 to 0x701e41 (0x28a91e bytes) 000041[0] 1766394343.236842 mapread 0x520c9a thru 0x522dd2 (0x2139 bytes) 000042[1] 1766394343.245489 write 0x2db16f thru 0x2e4aa9 (0x993b bytes) 000044[0] 1766394343.261020 mapwrite 0x1b06ab thru 0x1b47ce (0x4124 bytes) 000045[1] 1766394343.287540 trunc from 0x701e42 to 0x60daab (0xf4396 bytes) 000046[0] 1766394343.296851 trunc from 0x60daac to 0x7dfa2a (0x1d1f7f bytes) 000047[1] 1766394343.310758 read 0x199f69 thru 0x1a8ff7 (0xf08f bytes) 000049[0] 1766394343.321473 mapread 0x292cce thru 0x29d86e (0xaba1 bytes) 000051[0] 1766394343.330478 mapwrite 0x1eac6d thru 0x1f9932 (0xecc6 bytes) 000052[0] 1766394343.351622 mapread 0x1ed419 thru 0x1f9c58 (0xc840 bytes) 000053[1] 1766394343.352060 trunc from 0x7dfa2b to 0x10073f (0x6df2eb bytes) 000054[1] 1766394343.365922 write 0x67ce32 thru 0x68073d (0x390c bytes) 000055[0] 1766394343.372735 mapread 0x23f8b6 thru 0x242bcb (0x3316 bytes) 000056[1] 1766394343.392259 trunc from 0x68073e to 0x0c3c0d (0x5bcb30 bytes) 000058[0] 1766394343.404379 mapwrite 0x4b76da thru 0x4c1aca (0xa3f1 bytes) 000062[1] 1766394343.440943 write 0x9cf75f thru 0x9da3f6 (0xac98 bytes) 000064[0] 1766394343.452991 read 0x70962e thru 0x7165ef (0xcfc2 bytes) 000068[1] 1766394343.483121 read 0x16800d thru 0x17067e (0x8672 bytes) 000071[0] 1766394343.496024 read 0x40a574 thru 0x415bb2 (0xb63f bytes) 000072[0] 1766394343.501486 trunc from 0x9da3f7 to 0x153c09 (0x8867ed bytes) 000076[1] 1766394343.516812 mapwrite 0x124ae8 thru 0x1333a1 (0xe8ba bytes) 000077[1] 1766394343.547160 mapread 0x0e2899 thru 0x0f05bd (0xdd25 bytes) 000078[1] 1766394343.557018 read 0x14d388 thru 0x14fd46 (0x29bf bytes) 000080[1] 1766394343.573619 mapread 0x10c1ec thru 0x11c07f (0xfe94 bytes) 000081[0] 1766394343.578732 write 0x7393c6 thru 0x73d26a (0x3ea5 bytes) 000082[1] 1766394343.584166 trunc from 0x73d26b to 0x6b94f1 (0x83d79 bytes) 000083[1] 1766394343.593175 trunc from 0x6b94f2 to 0x7aac7c (0xf178b bytes) 000085[1] 1766394343.597611 mapwrite 0x5efd35 thru 0x5f9e51 (0xa11d bytes) 000087[1] 1766394343.645085 mapwrite 0x3639da thru 0x36da4d (0xa074 bytes) 000088[1] 1766394343.662568 mapread 0x1c1139 thru 0x1cf5b9 (0xe481 bytes) 000089[1] 1766394343.667082 mapwrite 0x7e8b7d thru 0x7ee84b (0x5ccf bytes) 000090[1] 1766394343.693583 write 0x68d0c6 thru 0x692213 (0x514e bytes) 000093[1] 1766394343.702958 mapread 0x5847cb thru 0x58e5b5 (0x9deb bytes) 000095[0] 1766394343.711470 write 0x403f5f thru 0x40644d (0x24ef bytes) 000096[1] 1766394343.725016 mapwrite 0x159785 thru 0x15baef (0x236b bytes) 000097[1] 1766394343.741227 mapwrite 0x98e606 thru 0x98f658 (0x1053 bytes) 000100[1] 1766394343.785349 mapread 0x3d822a thru 0x3d8b58 (0x92f bytes) 000101[1] 1766394343.785784 write 0x4d852a thru 0x4dc4ff (0x3fd6 bytes) 000102[1] 1766394343.788569 mapread 0x8e04d8 thru 0x8ebab8 (0xb5e1 bytes) 000103[0] 1766394343.793483 write 0x8dce38 thru 0x8e4200 (0x73c9 bytes) 000104[1] 1766394343.806390 trunc from 0x98f659 to 0x9c42e6 (0x34c8e bytes) 000109[0] 1766394343.830571 mapwrite 0x4a82a8 thru 0x4b0bb3 (0x890c bytes) 000111[1] 1766394343.859335 mapwrite 0x93a69f thru 0x94a43a (0xfd9c bytes) 000112[1] 1766394343.894801 trunc from 0x9c42e7 to 0x0e18c4 (0x8e2a22 bytes) 000114[0] 1766394343.902439 trunc from 0x0e18c5 to 0x4f7290 (0x4159cc bytes) 000115[1] 1766394343.916332 trunc from 0x4f7291 to 0x3a06ba (0x156bd6 bytes) 000116[1] 1766394343.924403 mapread 0x2820d1 thru 0x286f63 (0x4e93 bytes) 000117[1] 1766394343.936580 trunc from 0x3a06bb to 0x74b2f3 (0x3aac39 bytes) 000120[0] 1766394343.943320 mapread 0x35761e thru 0x367430 (0xfe13 bytes) 000121[0] 1766394343.958505 write 0x130e22 thru 0x136758 (0x5937 bytes) 000123[0] 1766394343.969374 trunc from 0x74b2f4 to 0x4fa53a (0x250db9 bytes) 000127[1] 1766394343.980987 trunc from 0x4fa53b to 0x693e4c (0x199912 bytes) 000128[0] 1766394343.996582 mapwrite 0x88a524 thru 0x88d260 (0x2d3d bytes) 000130[0] 1766394344.025710 write 0x8cc405 thru 0x8d3f41 (0x7b3d bytes) 000133[0] 1766394344.034515 mapread 0x0764ea thru 0x079e71 (0x3988 bytes) 000134[0] 1766394344.039379 read 0x030777 thru 0x0367a7 (0x6031 bytes) 000135[0] 1766394344.045970 read 0x38605c thru 0x391430 (0xb3d5 bytes) 000138[0] 1766394344.056552 read 0x3fd168 thru 0x4088cf (0xb768 bytes) 000148[1] 1766394344.078949 read 0x500b54 thru 0x510602 (0xfaaf bytes) 000149[0] 1766394344.084597 write 0x511f58 thru 0x51b03d (0x90e6 bytes) 000150[1] 1766394344.092986 mapwrite 0x27d07e thru 0x284d97 (0x7d1a bytes) 000156[1] 1766394344.135899 trunc from 0x8d3f42 to 0x47da98 (0x4564a9 bytes) 000157[0] 1766394344.147720 trunc from 0x47da99 to 0x3debdb (0x9eebd bytes) 000158[0] 1766394344.155686 read 0x3c637d thru 0x3c8960 (0x25e4 bytes) 000161[0] 1766394344.165993 write 0x0e1684 thru 0x0efa4d (0xe3ca bytes) 000162[1] 1766394344.168977 read 0x3434ac thru 0x34b918 (0x846d bytes) 000168[0] 1766394344.217066 trunc from 0x3debdc to 0x7545e5 (0x375a0a bytes) 000171[1] 1766394344.234321 mapread 0x1486c9 thru 0x15094b (0x8283 bytes) 000174[0] 1766394344.255919 trunc from 0x7545e6 to 0x6723f8 (0xe21ed bytes) 000175[1] 1766394344.263581 write 0x1d0b79 thru 0x1df31e (0xe7a6 bytes) 000176[1] 1766394344.273086 trunc from 0x6723f9 to 0x61d621 (0x54dd7 bytes) 000178[1] 1766394344.281307 mapread 0x328a8e thru 0x329775 (0xce8 bytes) 000180[0] 1766394344.293446 mapwrite 0x7b0811 thru 0x7b1a13 (0x1203 bytes) 000189[1] 1766394344.338420 trunc from 0x7b1a14 to 0x74e074 (0x6399f bytes) 000190[0] 1766394344.350323 mapread 0x73fedc thru 0x743ef4 (0x4019 bytes) 000191[0] 1766394344.360004 mapread 0x329e18 thru 0x32df14 (0x40fd bytes) 000193[1] 1766394344.371499 mapwrite 0x05d01f thru 0x0638cd (0x68af bytes) 000198[1] 1766394344.408991 write 0x742f72 thru 0x74bab9 (0x8b48 bytes) 000199[1] 1766394344.433759 trunc from 0x74e075 to 0x8a4795 (0x156721 bytes) 000200[1] 1766394344.442908 mapwrite 0x094261 thru 0x09d964 (0x9704 bytes) 000203[1] 1766394344.476508 read 0x5875d2 thru 0x594cdd (0xd70c bytes) 000207[0] 1766394344.486705 mapwrite 0x84c1bc thru 0x8525ec (0x6431 bytes) 000212[1] 1766394344.524367 mapwrite 0x5e8ca2 thru 0x5f0824 (0x7b83 bytes) 000214[0] 1766394344.546525 read 0x463a37 thru 0x464a3b (0x1005 bytes) 000219[0] 1766394344.566761 mapread 0x83e2c2 thru 0x83f0f5 (0xe34 bytes) 000220[0] 1766394344.572298 mapread 0x0e23b6 thru 0x0efb0b (0xd756 bytes) 000222[1] 1766394344.584495 trunc from 0x8a4796 to 0x21b096 (0x6896ff bytes) 000224[1] 1766394344.601412 read 0x18997d thru 0x1952ca (0xb94e bytes) 000227[0] 1766394344.623991 mapread 0x07ecca thru 0x0868b6 (0x7bed bytes) 000230[0] 1766394344.639471 mapwrite 0x65a748 thru 0x664fac (0xa865 bytes) 000232[1] 1766394344.665799 mapwrite 0x739ff1 thru 0x749be4 (0xfbf4 bytes) 000236[0] 1766394344.726507 read 0x04b26b thru 0x04b313 (0xa9 bytes) 000237[0] 1766394344.732760 trunc from 0x749be5 to 0x533366 (0x21687e bytes) 000239[1] 1766394344.749213 mapread 0x0af828 thru 0x0b093c (0x1115 bytes) 000240[0] 1766394344.758124 read 0x182507 thru 0x18bc08 (0x9702 bytes) 000245[1] 1766394344.762498 mapwrite 0x1f86d1 thru 0x1f8969 (0x299 bytes) 000246[1] 1766394344.779985 mapread 0x4adfe2 thru 0x4b0ae2 (0x2b01 bytes) 000248[1] 1766394344.788184 mapwrite 0x35a8e7 thru 0x36473f (0x9e59 bytes) 000251[1] 1766394344.816746 write 0x807bda thru 0x80e289 (0x66b0 bytes) 000252[0] 1766394344.818225 trunc from 0x80e28a to 0x58dc8a (0x2805ff bytes) 000253[1] 1766394344.830804 mapwrite 0x28937e thru 0x28ed04 (0x5987 bytes) 000254[0] 1766394344.857599 mapread 0x0a993c thru 0x0ae99c (0x5061 bytes) 000256[1] 1766394344.866389 read 0x381337 thru 0x382f98 (0x1c62 bytes) 000258[1] 1766394344.873402 mapread 0x3b78a6 thru 0x3bfc82 (0x83dd bytes) 000259[0] 1766394344.883116 mapread 0x4ba344 thru 0x4c55fc (0xb2b9 bytes) 000262[0] 1766394344.890662 write 0x86e19a thru 0x875c32 (0x7a99 bytes) 000263[1] 1766394344.895295 write 0x0f4320 thru 0x0f6e39 (0x2b1a bytes) 000264[0] 1766394344.905419 mapwrite 0x327a05 thru 0x333651 (0xbc4d bytes) 000265[0] 1766394344.932744 write 0x0b9d57 thru 0x0bfd00 (0x5faa bytes) 000268[0] 1766394344.939987 mapread 0x020b1c thru 0x02b943 (0xae28 bytes) 000269[1] 1766394344.944872 write 0x30207a thru 0x304307 (0x228e bytes) 000270[0] 1766394344.955490 mapread 0x649f5f thru 0x651032 (0x70d4 bytes) 000274[0] 1766394344.997365 trunc from 0x875c33 to 0x474fc2 (0x400c70 bytes) 000280[1] 1766394345.018609 mapwrite 0x315061 thru 0x322d1f (0xdcbf bytes) 000281[0] 1766394345.043966 read 0x13a725 thru 0x13dd0b (0x35e7 bytes) 000282[1] 1766394345.047871 mapwrite 0x3c3d2e thru 0x3d2fa7 (0xf27a bytes) 000283[0] 1766394345.074013 mapwrite 0x9db9f3 thru 0x9e65b6 (0xabc4 bytes) 000285[0] 1766394345.102658 mapread 0x23f62a thru 0x24c9c2 (0xd399 bytes) 000289[0] 1766394345.119127 mapread 0x37c8fa thru 0x388093 (0xb79a bytes) 000291[0] 1766394345.129278 mapread 0x37826a thru 0x3875ec (0xf383 bytes) 000292[0] 1766394345.129844 mapread 0x956b12 thru 0x965078 (0xe567 bytes) 000293[0] 1766394345.141308 read 0x422117 thru 0x42de78 (0xbd62 bytes) 000294[1] 1766394345.147647 read 0x390f66 thru 0x395daf (0x4e4a bytes) 000295[0] 1766394345.153573 read 0x9d8790 thru 0x9dcb19 (0x438a bytes) 000296[0] 1766394345.155981 read 0x129324 thru 0x133dba (0xaa97 bytes) 000297[0] 1766394345.161926 mapwrite 0x662f57 thru 0x6642e4 (0x138e bytes) 000298[0] 1766394345.177344 mapread 0x8e99d9 thru 0x8f75d1 (0xdbf9 bytes) 000301[1] 1766394345.190697 read 0x3460f1 thru 0x346127 (0x37 bytes) 000304[1] 1766394345.198813 trunc from 0x9e65b7 to 0x394545 (0x652071 bytes) 000306[0] 1766394345.212436 write 0x8a28de thru 0x8ae92d (0xc050 bytes) 000309[0] 1766394345.226197 mapread 0x85deb3 thru 0x86dcf9 (0xfe47 bytes) 000311[0] 1766394345.238086 trunc from 0x8ae92e to 0x9e3c52 (0x135325 bytes) 000312[0] 1766394345.245051 mapread 0x8bd37a thru 0x8c26fe (0x5385 bytes) 000313[1] 1766394345.252229 mapwrite 0x609e5e thru 0x6148e0 (0xaa83 bytes) 000316[1] 1766394345.288295 write 0x55f2b1 thru 0x56c6eb (0xd43b bytes) 000317[1] 1766394345.298144 write 0x5ab745 thru 0x5b560a (0x9ec6 bytes) 000319[0] 1766394345.307877 read 0x851932 thru 0x85aecc (0x959b bytes) 000323[0] 1766394345.322944 read 0x7c1282 thru 0x7d03d0 (0xf14f bytes) 000324[1] 1766394345.331527 write 0x1ba7f2 thru 0x1c9d1c (0xf52b bytes) 000325[1] 1766394345.339267 write 0x39d3d7 thru 0x3a3d13 (0x693d bytes) 000327[0] 1766394345.345062 read 0x019539 thru 0x01d937 (0x43ff bytes) 000329[0] 1766394345.352625 mapwrite 0x37552d thru 0x375859 (0x32d bytes) 000330[0] 1766394345.371680 trunc from 0x9e3c53 to 0x0414ae (0x9a27a4 bytes) 000334[1] 1766394345.389215 write 0x96ec5e thru 0x972c50 (0x3ff3 bytes) 000335[0] 1766394345.397296 mapread 0x4b8349 thru 0x4c0e60 (0x8b18 bytes) 000340[0] 1766394345.443160 write 0x1aaded thru 0x1ac064 (0x1278 bytes) 000347[0] 1766394345.466982 read 0x3a5a1a thru 0x3a5a86 (0x6d bytes) 000349[1] 1766394345.476493 mapwrite 0x40fa1f thru 0x41af4c (0xb52e bytes) 000350[0] 1766394345.552452 read 0x740c91 thru 0x74bac4 (0xae34 bytes) 000353[0] 1766394345.569577 trunc from 0x972c51 to 0x43637b (0x53c8d5 bytes) 000354[1] 1766394345.580009 mapwrite 0x960f04 thru 0x967304 (0x6401 bytes) 000355[0] 1766394345.611335 trunc from 0x967305 to 0x0ea0b2 (0x87d252 bytes) 000357[0] 1766394345.629611 read 0x0925f8 thru 0x095efc (0x3905 bytes) 000362[0] 1766394345.654615 read 0x01338b thru 0x02094c (0xd5c2 bytes) 000365[0] 1766394345.659570 write 0x4df869 thru 0x4e732b (0x7ac3 bytes) 000368[0] 1766394345.665389 mapwrite 0x4e1338 thru 0x4f06d1 (0xf39a bytes) 000371[0] 1766394345.704272 read 0x24a33c thru 0x251fe8 (0x7cad bytes) 000372[0] 1766394345.704586 mapwrite 0x35533b thru 0x356006 (0xccc bytes) 000373[1] 1766394345.719967 write 0x4391b2 thru 0x43b88f (0x26de bytes) 000374[1] 1766394345.728625 mapread 0x1f6563 thru 0x1fc413 (0x5eb1 bytes) 000375[1] 1766394345.744473 trunc from 0x4f06d2 to 0x95e0f2 (0x46da21 bytes) 000377[0] 1766394345.757251 write 0x2108af thru 0x21c501 (0xbc53 bytes) 000378[0] 1766394345.764876 trunc from 0x95e0f3 to 0x47e2fd (0x4dfdf5 bytes) 000379[1] 1766394345.772706 read 0x14568d thru 0x147ed5 (0x2849 bytes) 000381[0] 1766394345.783725 read 0x04ae3f thru 0x04b704 (0x8c6 bytes) 000382[0] 1766394345.785402 trunc from 0x47e2fe to 0x2a6962 (0x1d799b bytes) 000383[0] 1766394345.792219 mapwrite 0x6877bd thru 0x69368e (0xbed2 bytes) 000384[1] 1766394345.815278 read 0x5494fc thru 0x5526f4 (0x91f9 bytes) 000386[0] 1766394345.825706 mapwrite 0x3c3b71 thru 0x3d29d1 (0xee61 bytes) 000388[1] 1766394345.848949 write 0x39463b thru 0x39ffda (0xb9a0 bytes) 000390[1] 1766394345.857377 mapread 0x283c12 thru 0x28a3f2 (0x67e1 bytes) 000394[0] 1766394345.872283 read 0x65c578 thru 0x663ed7 (0x7960 bytes) 000395[0] 1766394345.874838 write 0x48eddc thru 0x49d740 (0xe965 bytes) 000397[0] 1766394345.886667 mapread 0x44d0e1 thru 0x44da86 (0x9a6 bytes) 000398[0] 1766394345.893877 write 0x551820 thru 0x55c0a3 (0xa884 bytes) 000400[1] 1766394345.903586 trunc from 0x69368f to 0x9b4131 (0x320aa3 bytes) 000404[1] 1766394345.926993 mapwrite 0x58b16b thru 0x58cadd (0x1973 bytes) 000407[1] 1766394345.942723 write 0x3ca406 thru 0x3d6839 (0xc434 bytes) 000408[1] 1766394345.951241 write 0x5a2379 thru 0x5b1a39 (0xf6c1 bytes) 000410[1] 1766394345.960160 write 0x3ad5c4 thru 0x3b8054 (0xaa91 bytes) 000412[1] 1766394345.963876 trunc from 0x9b4132 to 0x094f15 (0x91f21c bytes) 000414[1] 1766394345.976702 read 0x08fa21 thru 0x094f15 (0x54f5 bytes) 000417[0] 1766394345.983712 write 0x065526 thru 0x067e9a (0x2975 bytes) 000422[0] 1766394346.006142 mapwrite 0x54baec thru 0x556ae2 (0xaff7 bytes) 000428[0] 1766394346.049243 read 0x54f7c7 thru 0x54f896 (0xd0 bytes) 000429[1] 1766394346.049604 write 0x762f2f thru 0x768a3b (0x5b0d bytes) 000432[1] 1766394346.061715 trunc from 0x768a3c to 0x1fd8f5 (0x56b146 bytes) 000434[0] 1766394346.077382 trunc from 0x1fd8f6 to 0x9631f1 (0x7658fc bytes) 000441[0] 1766394346.099655 read 0x5a8c1f thru 0x5b741d (0xe7ff bytes) 000444[1] 1766394346.113485 trunc from 0x9631f2 to 0x9a26db (0x3f4ea bytes) 000446[0] 1766394346.126518 mapwrite 0x506694 thru 0x50f6f0 (0x905d bytes) 000448[0] 1766394346.153852 mapread 0x21bf2a thru 0x21e73f (0x2816 bytes) 000450[1] 1766394346.164524 mapwrite 0x88d1c4 thru 0x8935a2 (0x63df bytes) 000452[1] 1766394346.219826 write 0x9e60a4 thru 0x9f3676 (0xd5d3 bytes) 000453[1] 1766394346.224053 mapread 0x7f23c3 thru 0x7ff71b (0xd359 bytes) 000454[0] 1766394346.254259 read 0x4b1a13 thru 0x4b2a88 (0x1076 bytes) 000455[1] 1766394346.256906 mapwrite 0x7e198f thru 0x7eab4e (0x91c0 bytes) 000458[0] 1766394346.306394 read 0x222c58 thru 0x230925 (0xdcce bytes) 000459[0] 1766394346.316403 write 0x3178b8 thru 0x325099 (0xd7e2 bytes) 000461[1] 1766394346.329802 write 0x6bf811 thru 0x6cba03 (0xc1f3 bytes) 000463[1] 1766394346.338759 mapread 0x661656 thru 0x6693c8 (0x7d73 bytes) 000464[0] 1766394346.352318 write 0x425158 thru 0x4288b6 (0x375f bytes) 000465[0] 1766394346.361922 mapwrite 0x6470c5 thru 0x64a554 (0x3490 bytes) 000466[0] 1766394346.381367 mapwrite 0x24d16a thru 0x24f165 (0x1ffc bytes) 000467[1] 1766394346.398817 trunc from 0x9f3677 to 0x505c7a (0x4ed9fc bytes) 000471[1] 1766394346.414490 read 0x476228 thru 0x48082f (0xa608 bytes) 000472[0] 1766394346.417083 mapwrite 0x81ddb8 thru 0x829b06 (0xbd4f bytes) 000473[0] 1766394346.445115 trunc from 0x829b07 to 0x8c84d4 (0x9e9ce bytes) 000474[1] 1766394346.452592 read 0x26d12f thru 0x27209e (0x4f70 bytes) 000476[1] 1766394346.464078 write 0x26ab11 thru 0x27180e (0x6cfe bytes) 000477[0] 1766394346.472312 trunc from 0x8c84d5 to 0x777b38 (0x15099c bytes) 000478[0] 1766394346.485275 read 0x750f6c thru 0x75ad3b (0x9dd0 bytes) 000479[0] 1766394346.491737 mapread 0x4eed8f thru 0x4f6285 (0x74f7 bytes) 000480[0] 1766394346.497661 mapwrite 0x37857f thru 0x380111 (0x7b93 bytes) 000482[1] 1766394346.515730 mapwrite 0x82da78 thru 0x83c1f6 (0xe77f bytes) 000483[0] 1766394346.554592 read 0x7e1a4d thru 0x7ef5cc (0xdb80 bytes) 000484[1] 1766394346.561855 trunc from 0x83c1f7 to 0x3ef31e (0x44ced8 bytes) 000485[0] 1766394346.568985 mapread 0x04c12f thru 0x058c9b (0xcb6d bytes) 000487[0] 1766394346.585213 trunc from 0x3ef31f to 0x0e43a8 (0x30af76 bytes) 000488[0] 1766394346.598569 trunc from 0x0e43a9 to 0x25e50b (0x17a163 bytes) 000491[1] 1766394346.612076 read 0x184a99 thru 0x187981 (0x2ee9 bytes) 000492[1] 1766394346.635876 mapread 0x199fe1 thru 0x19b2e9 (0x1309 bytes) 000493[0] 1766394346.640603 read 0x0ed374 thru 0x0f39e5 (0x6672 bytes) 000494[1] 1766394346.654615 read 0x099605 thru 0x0a1fc7 (0x89c3 bytes) 000495[1] 1766394346.658047 mapwrite 0x867d91 thru 0x87135f (0x95cf bytes) 000496[0] 1766394346.741813 write 0x228fd3 thru 0x238034 (0xf062 bytes) 000500[1] 1766394346.778055 trunc from 0x871360 to 0x719509 (0x157e56 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4346 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1766394347.052360 mapwrite 0x313000 thru 0x321284 (0xe285 bytes) 000002[0] 1766394347.091717 mapwrite 0x032000 thru 0x03ad56 (0x8d57 bytes) 000004[1] 1766394347.113770 mapwrite 0x691000 thru 0x69e8cc (0xd8cd bytes) 000005[1] 1766394347.146215 read 0x2d7000 thru 0x2dd4d3 (0x64d4 bytes) 000010[1] 1766394347.154396 mapwrite 0x300000 thru 0x30cffa (0xcffb bytes) 000011[1] 1766394347.172400 read 0x29b000 thru 0x2a6693 (0xb694 bytes) 000012[1] 1766394347.174919 write 0x6d5000 thru 0x6da438 (0x5439 bytes) 000014[0] 1766394347.178885 read 0x001000 thru 0x0063bf (0x53c0 bytes) 000015[0] 1766394347.186035 write 0x3ad000 thru 0x3b8c85 (0xbc86 bytes) 000022[1] 1766394347.213527 read 0x2fd000 thru 0x300877 (0x3878 bytes) 000028[1] 1766394347.221584 mapread 0x61f000 thru 0x620378 (0x1379 bytes) 000030[1] 1766394347.224793 read 0x018000 thru 0x01d698 (0x5699 bytes) 000032[0] 1766394347.227421 mapread 0x0bf000 thru 0x0c08e0 (0x18e1 bytes) 000034[0] 1766394347.230170 mapwrite 0x224000 thru 0x233d0b (0xfd0c bytes) 000035[0] 1766394347.257972 mapwrite 0x821000 thru 0x82ae7a (0x9e7b bytes) 000036[1] 1766394347.283638 mapread 0x2a2000 thru 0x2b1760 (0xf761 bytes) 000037[0] 1766394347.294493 write 0x4b3000 thru 0x4bc9f9 (0x99fa bytes) 000043[0] 1766394347.306516 read 0x4da000 thru 0x4e76e5 (0xd6e6 bytes) truncating to largest ever: 0x80d243 000046[1] 1766394347.313086 trunc from 0x82ae7b to 0x80d242 (0x1dc38 bytes) 000050[1] 1766394347.335012 mapread 0x210000 thru 0x216a93 (0x6a94 bytes) 000051[1] 1766394347.337938 mapwrite 0x102000 thru 0x102bed (0xbee bytes) 000052[1] 1766394347.349914 mapwrite 0x9b2000 thru 0x9c1a88 (0xfa89 bytes) 000059[1] 1766394347.388739 mapread 0x475000 thru 0x483745 (0xe746 bytes) 000061[1] 1766394347.395936 mapread 0x654000 thru 0x6630f2 (0xf0f3 bytes) 000063[0] 1766394347.406187 read 0x95c000 thru 0x95ff51 (0x3f52 bytes) 000067[1] 1766394347.410477 mapread 0x977000 thru 0x983e6f (0xce70 bytes) 000069[0] 1766394347.418629 mapread 0x406000 thru 0x4149cc (0xe9cd bytes) 000074[0] 1766394347.427400 mapread 0x888000 thru 0x88d057 (0x5058 bytes) 000075[0] 1766394347.432214 write 0x7ba000 thru 0x7c7cf2 (0xdcf3 bytes) 000076[1] 1766394347.436836 write 0x05c000 thru 0x069777 (0xd778 bytes) 000078[1] 1766394347.445784 write 0x50c000 thru 0x517e20 (0xbe21 bytes) 000079[0] 1766394347.447621 mapwrite 0x6e6000 thru 0x6f0122 (0xa123 bytes) 000081[0] 1766394347.478793 mapwrite 0x074000 thru 0x074197 (0x198 bytes) 000082[0] 1766394347.493846 trunc from 0x9c1a89 to 0x68ffc3 (0x331ac5 bytes) 000083[0] 1766394347.502571 write 0x4d2000 thru 0x4d32b1 (0x12b2 bytes) 000085[1] 1766394347.505659 read 0x684000 thru 0x6854e7 (0x14e8 bytes) 000086[0] 1766394347.510723 write 0x568000 thru 0x56a734 (0x2735 bytes) 000089[0] 1766394347.522660 mapwrite 0x867000 thru 0x870643 (0x9644 bytes) 000091[0] 1766394347.548236 write 0x2c8000 thru 0x2d2a5a (0xaa5b bytes) 000094[0] 1766394347.555195 mapwrite 0x533000 thru 0x539562 (0x6563 bytes) 000095[0] 1766394347.571672 write 0x044000 thru 0x05337c (0xf37d bytes) 000096[1] 1766394347.573905 write 0x837000 thru 0x842d0e (0xbd0f bytes) 000101[0] 1766394347.591220 read 0x1a6000 thru 0x1ae9e7 (0x89e8 bytes) 000103[0] 1766394347.598358 mapwrite 0x575000 thru 0x57dca2 (0x8ca3 bytes) 000105[1] 1766394347.619483 read 0x12c000 thru 0x1392b0 (0xd2b1 bytes) 000107[0] 1766394347.625760 mapwrite 0x646000 thru 0x65437b (0xe37c bytes) 000111[0] 1766394347.645640 mapread 0x048000 thru 0x04f60a (0x760b bytes) 000114[1] 1766394347.663074 mapwrite 0x0e5000 thru 0x0ea8a2 (0x58a3 bytes) 000115[1] 1766394347.688682 read 0x6b4000 thru 0x6bdcc3 (0x9cc4 bytes) 000118[0] 1766394347.691685 write 0x740000 thru 0x7461d2 (0x61d3 bytes) 000120[0] 1766394347.702293 read 0x1ff000 thru 0x2019f8 (0x29f9 bytes) 000122[1] 1766394347.704203 write 0x8e6000 thru 0x8f5681 (0xf682 bytes) 000123[1] 1766394347.709626 trunc from 0x8f5682 to 0x6bb3a7 (0x23a2da bytes) 000125[0] 1766394347.719063 write 0x6da000 thru 0x6dbdbf (0x1dc0 bytes) 000130[1] 1766394347.725250 write 0x5bf000 thru 0x5c74f8 (0x84f9 bytes) 000131[1] 1766394347.730171 trunc from 0x6dbdc0 to 0x678636 (0x63789 bytes) 000132[1] 1766394347.738059 read 0x026000 thru 0x02eb23 (0x8b24 bytes) 000133[1] 1766394347.742250 write 0x74a000 thru 0x74a279 (0x27a bytes) 000134[1] 1766394347.742989 mapwrite 0x84a000 thru 0x84bdb3 (0x1db4 bytes) 000136[0] 1766394347.770614 trunc from 0x84bdb4 to 0x1ada69 (0x69e34a bytes) 000142[1] 1766394347.785435 read 0x030000 thru 0x03918e (0x918f bytes) 000143[0] 1766394347.800614 mapread 0x18d000 thru 0x18f567 (0x2568 bytes) 000147[0] 1766394347.807520 write 0x547000 thru 0x556ac8 (0xfac9 bytes) 000149[0] 1766394347.820121 mapwrite 0x7af000 thru 0x7b50ca (0x60cb bytes) 000150[1] 1766394347.845024 read 0x300000 thru 0x3072fa (0x72fb bytes) 000153[1] 1766394347.853192 read 0x500000 thru 0x50ee20 (0xee21 bytes) 000154[1] 1766394347.856324 read 0x06f000 thru 0x0780d6 (0x90d7 bytes) 000155[1] 1766394347.859160 read 0x43b000 thru 0x44258b (0x758c bytes) 000156[1] 1766394347.861804 read 0x21a000 thru 0x224385 (0xa386 bytes) 000157[0] 1766394347.865043 trunc from 0x7b50cb to 0x1a249e (0x612c2c bytes) 000158[0] 1766394347.873682 read 0x0d2000 thru 0x0dac11 (0x8c12 bytes) 000170[1] 1766394347.910983 read 0x02b000 thru 0x02ce29 (0x1e2a bytes) 000171[0] 1766394347.925730 trunc from 0x1a249f to 0x02d71c (0x174d82 bytes) 000174[0] 1766394347.938960 write 0x6ad000 thru 0x6ba289 (0xd28a bytes) 000176[1] 1766394347.941414 mapread 0x44f000 thru 0x45aec7 (0xbec8 bytes) 000178[0] 1766394347.977810 read 0x2ce000 thru 0x2d80f4 (0xa0f5 bytes) 000179[1] 1766394347.980613 mapread 0x3ba000 thru 0x3bb683 (0x1684 bytes) 000180[1] 1766394347.981087 mapwrite 0x433000 thru 0x439fd4 (0x6fd5 bytes) truncating to largest ever: 0x8e4a5b 000182[0] 1766394348.026653 trunc from 0x6ba28a to 0x8e4a5a (0x22a7d1 bytes) 000185[1] 1766394348.044205 trunc from 0x8e4a5b to 0x350917 (0x594143 bytes) 000188[0] 1766394348.057596 mapwrite 0x9e7000 thru 0x9eea8c (0x7a8d bytes) 000189[0] 1766394348.083422 write 0x94f000 thru 0x955805 (0x6806 bytes) 000190[0] 1766394348.090180 read 0x723000 thru 0x726744 (0x3745 bytes) 000193[1] 1766394348.095599 trunc from 0x9eea8d to 0x38dcd3 (0x660db9 bytes) 000196[1] 1766394348.109043 write 0x83f000 thru 0x8421c4 (0x31c5 bytes) 000197[0] 1766394348.114162 trunc from 0x8421c5 to 0x8e031e (0x9e15a bytes) 000199[0] 1766394348.122804 read 0x772000 thru 0x780acb (0xeacc bytes) 000200[0] 1766394348.126031 mapread 0x331000 thru 0x336c96 (0x5c97 bytes) 000203[0] 1766394348.144507 read 0x27c000 thru 0x2839ae (0x79af bytes) 000204[0] 1766394348.147002 write 0x347000 thru 0x34d127 (0x6128 bytes) 000205[1] 1766394348.151399 read_OD 0x8b2000 thru 0x8b4fff (0x3000 bytes) 000206[1] 1766394348.160161 mapread 0x8b3000 thru 0x8b7350 (0x4351 bytes) 000207[0] 1766394348.166553 write 0x013000 thru 0x01edf5 (0xbdf6 bytes) 000209[0] 1766394348.175632 read 0x4e1000 thru 0x4e6ed1 (0x5ed2 bytes) 000215[1] 1766394348.187081 read_OD 0x3a3000 thru 0x3b1fff (0xf000 bytes) 000216[0] 1766394348.195739 mapread 0x1aa000 thru 0x1b2683 (0x8684 bytes) 000218[1] 1766394348.201670 read_OD 0x52a000 thru 0x538fff (0xf000 bytes) 000219[1] 1766394348.204029 mapread 0x85d000 thru 0x8665c0 (0x95c1 bytes) 000220[1] 1766394348.208829 mapwrite 0x80a000 thru 0x8185e8 (0xe5e9 bytes) 000224[0] 1766394348.240061 mapread 0x640000 thru 0x64e1c1 (0xe1c2 bytes) 000225[1] 1766394348.252969 read_OD 0x095000 thru 0x095fff (0x1000 bytes) 000229[0] 1766394348.255149 mapwrite 0x465000 thru 0x46c772 (0x7773 bytes) 000231[0] 1766394348.278338 write 0x392000 thru 0x39a6f6 (0x86f7 bytes) 000232[1] 1766394348.283685 read_OD 0x2f6000 thru 0x2fbfff (0x6000 bytes) 000234[1] 1766394348.293438 mapwrite 0x6a4000 thru 0x6ab50f (0x7510 bytes) 000235[1] 1766394348.320453 trunc from 0x8e031f to 0x1e983f (0x6f6adf bytes) 000237[1] 1766394348.330808 trunc from 0x1e9840 to 0x67157a (0x487d3b bytes) 000238[1] 1766394348.341902 read_OD 0x275000 thru 0x277fff (0x3000 bytes) 000239[0] 1766394348.346288 write 0x56a000 thru 0x575d83 (0xbd84 bytes) 000241[0] 1766394348.350743 mapread 0x381000 thru 0x38445e (0x345f bytes) 000244[1] 1766394348.362401 read_OD 0x13f000 thru 0x149fff (0xb000 bytes) 000246[1] 1766394348.371013 write_OD 0x158000 thru 0x161fff (0xa000 bytes) 000248[1] 1766394348.385472 mapread 0x66a000 thru 0x671437 (0x7438 bytes) 000250[1] 1766394348.390160 mapwrite 0x156000 thru 0x15755f (0x1560 bytes) 000253[0] 1766394348.411635 mapread 0x377000 thru 0x37f66d (0x866e bytes) 000254[0] 1766394348.417736 trunc from 0x67157b to 0x6ba9bb (0x49441 bytes) 000255[1] 1766394348.425081 mapwrite 0x967000 thru 0x975ebf (0xeec0 bytes) 000256[0] 1766394348.451193 read 0x88e000 thru 0x89444b (0x644c bytes) 000257[1] 1766394348.456780 read_OD 0x550000 thru 0x559fff (0xa000 bytes) 000258[1] 1766394348.459963 write_OD 0x6af000 thru 0x6bbfff (0xd000 bytes) 000263[0] 1766394348.472977 write 0x131000 thru 0x13cf1e (0xbf1f bytes) 000264[1] 1766394348.480197 read_OD 0x021000 thru 0x022fff (0x2000 bytes) 000266[0] 1766394348.487916 write 0x41e000 thru 0x41f519 (0x151a bytes) 000267[1] 1766394348.494734 write_OD 0x49b000 thru 0x49cfff (0x2000 bytes) 000268[0] 1766394348.508132 mapread 0x52d000 thru 0x5304a4 (0x34a5 bytes) 000269[1] 1766394348.513784 read_OD 0x7d1000 thru 0x7d1fff (0x1000 bytes) 000271[0] 1766394348.515692 mapread 0x456000 thru 0x457fce (0x1fcf bytes) 000272[1] 1766394348.518062 mapread 0x151000 thru 0x15b8ae (0xa8af bytes) 000273[1] 1766394348.522487 trunc from 0x975ec0 to 0x17585b (0x800664 bytes) 000277[1] 1766394348.537416 mapwrite 0x5d5000 thru 0x5db20b (0x620c bytes) 000279[0] 1766394348.560233 mapread 0x3c0000 thru 0x3cb856 (0xb857 bytes) 000283[0] 1766394348.576252 mapread 0x1e6000 thru 0x1e7a26 (0x1a27 bytes) 000284[0] 1766394348.579002 write 0x33d000 thru 0x34460e (0x760f bytes) 000286[0] 1766394348.586902 mapwrite 0x33c000 thru 0x340f72 (0x4f73 bytes) 000290[0] 1766394348.608344 read 0x2f6000 thru 0x2f9f7d (0x3f7e bytes) 000291[0] 1766394348.610002 write 0x4c3000 thru 0x4c68ae (0x38af bytes) 000296[1] 1766394348.618459 mapread 0x0c3000 thru 0x0d1de9 (0xedea bytes) 000297[0] 1766394348.632445 mapwrite 0x682000 thru 0x68d277 (0xb278 bytes) 000300[1] 1766394348.668893 mapread 0x51d000 thru 0x51eb72 (0x1b73 bytes) 000301[1] 1766394348.677256 trunc from 0x68d278 to 0x369f26 (0x323351 bytes) 000307[1] 1766394348.692003 mapwrite 0x0d4000 thru 0x0de8dc (0xa8dd bytes) 000308[1] 1766394348.712756 write_OD 0x6aa000 thru 0x6abfff (0x2000 bytes) 000309[0] 1766394348.724554 write 0x1c5000 thru 0x1c6bdb (0x1bdc bytes) 000310[1] 1766394348.731374 read_OD 0x474000 thru 0x478fff (0x5000 bytes) 000311[1] 1766394348.739434 read_OD 0x59a000 thru 0x5a4fff (0xb000 bytes) 000314[0] 1766394348.743250 read 0x193000 thru 0x19cb40 (0x9b41 bytes) 000315[1] 1766394348.746353 mapread 0x6a4000 thru 0x6a7b19 (0x3b1a bytes) 000317[0] 1766394348.751451 mapwrite 0x46e000 thru 0x47c0f0 (0xe0f1 bytes) 000319[1] 1766394348.777736 trunc from 0x6ac000 to 0x5b82c4 (0xf3d3b bytes) 000320[1] 1766394348.788870 trunc from 0x5b82c5 to 0x11950b (0x49edb9 bytes) 000323[1] 1766394348.794776 read_OD 0x10f000 thru 0x110fff (0x2000 bytes) truncating to largest ever: 0x9c3efb 000327[1] 1766394348.801584 trunc from 0x11950c to 0x9c3efa (0x8aa9ef bytes) 000329[1] 1766394348.815576 read_OD 0x75d000 thru 0x763fff (0x7000 bytes) 000330[0] 1766394348.819690 mapread 0x764000 thru 0x76dbe9 (0x9bea bytes) 000342[1] 1766394348.827297 mapwrite 0x3ce000 thru 0x3d042e (0x242f bytes) 000349[0] 1766394348.868605 read 0x3db000 thru 0x3e9dfa (0xedfb bytes) 000350[1] 1766394348.875012 read_OD 0x3ca000 thru 0x3d8fff (0xf000 bytes) 000351[1] 1766394348.877660 trunc from 0x9c3efb to 0x2f8291 (0x6cbc69 bytes) 000353[0] 1766394348.890520 mapwrite 0x38e000 thru 0x398ce8 (0xace9 bytes) 000356[0] 1766394348.922087 mapread 0x2b7000 thru 0x2bdcec (0x6ced bytes) 000359[1] 1766394348.934355 read_OD 0x00e000 thru 0x017fff (0xa000 bytes) 000360[0] 1766394348.940350 read 0x1e4000 thru 0x1f1fa6 (0xdfa7 bytes) 000362[0] 1766394348.965012 trunc from 0x398ce9 to 0x983796 (0x5eaaae bytes) 000363[0] 1766394348.980339 mapwrite 0x843000 thru 0x84e2f7 (0xb2f8 bytes) 000364[0] 1766394349.041026 mapwrite 0x502000 thru 0x505bb2 (0x3bb3 bytes) 000365[0] 1766394349.052919 read 0x938000 thru 0x93e155 (0x6156 bytes) 000366[1] 1766394349.053355 mapwrite 0x289000 thru 0x29013d (0x713e bytes) 000367[1] 1766394349.109584 read_OD 0x2f8000 thru 0x2f8fff (0x1000 bytes) 000371[1] 1766394349.119199 read_OD 0x8ef000 thru 0x8f1fff (0x3000 bytes) 000374[1] 1766394349.121540 read_OD 0x446000 thru 0x44efff (0x9000 bytes) 000375[1] 1766394349.123908 read_OD 0x834000 thru 0x836fff (0x3000 bytes) 000382[1] 1766394349.127180 write_OD 0x9d0000 thru 0x9dcfff (0xd000 bytes) 000383[0] 1766394349.140531 mapwrite 0x87b000 thru 0x8890f7 (0xe0f8 bytes) 000385[0] 1766394349.169238 read 0x6c1000 thru 0x6ca167 (0x9168 bytes) 000386[0] 1766394349.174476 read 0x285000 thru 0x288c1b (0x3c1c bytes) 000388[0] 1766394349.176569 read 0x7cc000 thru 0x7cdf46 (0x1f47 bytes) 000391[0] 1766394349.178495 read 0x63d000 thru 0x64b2c2 (0xe2c3 bytes) 000392[1] 1766394349.184259 read_OD 0x35f000 thru 0x367fff (0x9000 bytes) 000394[0] 1766394349.191837 read 0x7ed000 thru 0x7f732b (0xa32c bytes) 000396[0] 1766394349.195362 mapread 0x8a5000 thru 0x8a517a (0x17b bytes) 000397[0] 1766394349.203053 read 0x9d0000 thru 0x9d2a0f (0x2a10 bytes) 000398[1] 1766394349.204951 write_OD 0x02a000 thru 0x030fff (0x7000 bytes) 000401[0] 1766394349.219838 mapread 0x378000 thru 0x37c091 (0x4092 bytes) 000403[0] 1766394349.225714 trunc from 0x9dd000 to 0x59fe7c (0x43d183 bytes) 000404[0] 1766394349.233825 trunc from 0x59fe7d to 0x097f04 (0x507f78 bytes) 000406[0] 1766394349.243034 mapwrite 0x58c000 thru 0x594420 (0x8421 bytes) 000413[0] 1766394349.270737 read 0x445000 thru 0x448bc5 (0x3bc6 bytes) 000414[1] 1766394349.272594 mapwrite 0x9bd000 thru 0x9c7d36 (0xad37 bytes) 000415[0] 1766394349.295428 mapread 0x3b9000 thru 0x3c70dd (0xe0de bytes) 000416[0] 1766394349.306807 read 0x826000 thru 0x82b034 (0x5035 bytes) 000417[0] 1766394349.310359 trunc from 0x9c7d37 to 0x4ca054 (0x4fdce2 bytes) 000421[0] 1766394349.329533 read 0x206000 thru 0x2077e0 (0x17e1 bytes) 000422[1] 1766394349.334628 mapread 0x438000 thru 0x43e9d2 (0x69d3 bytes) 000426[0] 1766394349.344662 mapread 0x3ba000 thru 0x3bd038 (0x3039 bytes) 000428[1] 1766394349.349536 mapwrite 0x15c000 thru 0x16a472 (0xe473 bytes) 000429[0] 1766394349.376214 trunc from 0x4ca055 to 0x695ea8 (0x1cbe54 bytes) 000430[0] 1766394349.386093 mapread 0x141000 thru 0x1448cb (0x38cc bytes) 000431[1] 1766394349.392064 write_OD 0x1ef000 thru 0x1f5fff (0x7000 bytes) 000432[0] 1766394349.404322 write 0x132000 thru 0x139c4c (0x7c4d bytes) 000435[1] 1766394349.417410 write_OD 0x243000 thru 0x246fff (0x4000 bytes) 000439[1] 1766394349.431406 read_OD 0x140000 thru 0x142fff (0x3000 bytes) 000441[0] 1766394349.434372 mapwrite 0x246000 thru 0x24a9cd (0x49ce bytes) 000442[1] 1766394349.455255 trunc from 0x695ea9 to 0x5f202d (0xa3e7b bytes) 000443[1] 1766394349.462283 trunc from 0x5f202e to 0x77675b (0x18472e bytes) 000446[1] 1766394349.473781 write_OD 0x23c000 thru 0x241fff (0x6000 bytes) 000447[1] 1766394349.486252 read_OD 0x046000 thru 0x053fff (0xe000 bytes) 000452[1] 1766394349.490033 write_OD 0x4ce000 thru 0x4d0fff (0x3000 bytes) 000453[0] 1766394349.501575 trunc from 0x77675c to 0x3d7f00 (0x39e85b bytes) 000454[0] 1766394349.507982 write 0x0b1000 thru 0x0b8a61 (0x7a62 bytes) 000456[0] 1766394349.516766 write 0x510000 thru 0x510938 (0x939 bytes) 000457[0] 1766394349.520904 mapread 0x487000 thru 0x4887a6 (0x17a7 bytes) 000459[0] 1766394349.530895 trunc from 0x510939 to 0x250a9b (0x2bfe9d bytes) 000461[0] 1766394349.540542 read 0x03e000 thru 0x03e4b5 (0x4b6 bytes) 000465[0] 1766394349.568497 mapread 0x065000 thru 0x06c486 (0x7487 bytes) 000466[0] 1766394349.572591 mapwrite 0x76a000 thru 0x76bb77 (0x1b78 bytes) 000467[1] 1766394349.626318 mapread 0x312000 thru 0x321a64 (0xfa65 bytes) 000472[0] 1766394349.643798 mapread 0x1d1000 thru 0x1e0f74 (0xff75 bytes) 000473[1] 1766394349.663174 mapread 0x244000 thru 0x244915 (0x916 bytes) 000476[1] 1766394349.665791 write_OD 0x1f2000 thru 0x1fbfff (0xa000 bytes) 000478[0] 1766394349.678518 read 0x0e4000 thru 0x0f0414 (0xc415 bytes) 000480[1] 1766394349.681887 write_OD 0x817000 thru 0x823fff (0xd000 bytes) 000481[1] 1766394349.695227 mapwrite 0x431000 thru 0x43ca4c (0xba4d bytes) 000483[1] 1766394349.725142 mapread 0x0d2000 thru 0x0daa25 (0x8a26 bytes) 000484[0] 1766394349.732612 trunc from 0x824000 to 0x0ba897 (0x769768 bytes) 000487[0] 1766394349.743930 trunc from 0x0ba898 to 0x83de48 (0x7835b1 bytes) 000490[0] 1766394349.757203 write 0x438000 thru 0x43b91a (0x391b bytes) 000491[0] 1766394349.761993 mapwrite 0x629000 thru 0x633fe9 (0xafea bytes) 000497[1] 1766394349.790871 read_OD 0x2ba000 thru 0x2c1fff (0x8000 bytes) All operations completed A-OK! PASS 16a (10s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:05:51 (1766394351) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0926566 s, 90.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0596197 s, 141 MB/s Chance of close/open is 1 in 50 Seed set to 4351 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x15bd4d 000004[1] 1766394352.187004 trunc from 00000000 to 0x15bd4c (0x15bd4d bytes) truncating to largest ever: 0x4bad1b 000005[0] 1766394352.195330 trunc from 0x15bd4d to 0x4bad1a (0x35efce bytes) 000007[0] 1766394352.208876 mapread 0x2210f2 thru 0x221c05 (0xb14 bytes) 000011[0] 1766394352.218088 mapwrite 0x5ee0c7 thru 0x5ef016 (0xf50 bytes) 000012[1] 1766394352.239125 mapread 0x1f2731 thru 0x1f2c5c (0x52c bytes) 000013[1] 1766394352.246480 mapread 0x16f5f8 thru 0x16fa91 (0x49a bytes) 000014[1] 1766394352.252954 mapread 0x4e25b0 thru 0x4e3eef (0x1940 bytes) 000018[1] 1766394352.260941 mapread 0x17cb77 thru 0x17ce58 (0x2e2 bytes) 000020[1] 1766394352.263090 mapwrite 0x555018 thru 0x55620a (0x11f3 bytes) 000021[1] 1766394352.279557 write 0x26fe0e thru 0x270efe (0x10f1 bytes) 000022[1] 1766394352.288672 write 0x1a98b6 thru 0x1a9ecf (0x61a bytes) 000024[1] 1766394352.299240 mapwrite 0x788547 thru 0x789e39 (0x18f3 bytes) 000025[0] 1766394352.319387 write 0x619d8e thru 0x61bcf3 (0x1f66 bytes) 000033[1] 1766394352.355008 read 0x3f21c6 thru 0x3f24b0 (0x2eb bytes) 000034[0] 1766394352.359725 trunc from 0x789e3a to 0x01738c (0x772aad bytes) 000036[1] 1766394352.368468 trunc from 0x01738d to 0x481d3d (0x46a9b1 bytes) 000040[1] 1766394352.389289 mapread 0x0bba94 thru 0x0bd803 (0x1d70 bytes) truncating to largest ever: 0x8e374f 000041[0] 1766394352.393840 trunc from 0x481d3e to 0x8e374e (0x461a11 bytes) 000042[1] 1766394352.403410 mapread 0x3eed5b thru 0x3f0059 (0x12ff bytes) 000043[1] 1766394352.415192 write 0x046a32 thru 0x04774d (0xd1c bytes) 000044[1] 1766394352.422322 mapread 0x5414af thru 0x542aed (0x163f bytes) 000049[1] 1766394352.431665 read 0x8a7fe6 thru 0x8a821e (0x239 bytes) 000050[0] 1766394352.435968 trunc from 0x8e374f to 0x683005 (0x260749 bytes) 000051[1] 1766394352.445586 mapwrite 0x4d2956 thru 0x4d3905 (0xfb0 bytes) 000052[0] 1766394352.463779 read 0x0d4217 thru 0x0d59f7 (0x17e1 bytes) 000054[1] 1766394352.471710 mapwrite 0x3740b0 thru 0x375a8f (0x19e0 bytes) 000055[1] 1766394352.491448 trunc from 0x683006 to 0x27b7c1 (0x407844 bytes) 000056[1] 1766394352.499603 mapwrite 0x3c38ab thru 0x3c579b (0x1ef1 bytes) 000059[1] 1766394352.526941 write 0x355ab8 thru 0x356cb0 (0x11f9 bytes) 000060[0] 1766394352.532775 mapwrite 0x0aaf79 thru 0x0ab265 (0x2ed bytes) 000061[1] 1766394352.552885 mapwrite 0x754500 thru 0x75559d (0x109e bytes) 000062[1] 1766394352.571997 trunc from 0x75559e to 0x13a384 (0x61b219 bytes) 000063[1] 1766394352.575798 mapwrite 0x9209cb thru 0x92115e (0x794 bytes) 000064[0] 1766394352.597042 write 0x9f968a thru 0x9fa726 (0x109d bytes) 000066[1] 1766394352.605893 trunc from 0x9fa727 to 0x5854c8 (0x47525e bytes) truncating to largest ever: 0x9c0064 000069[1] 1766394352.623247 trunc from 0x5854c9 to 0x9c0063 (0x43ab9b bytes) 000070[0] 1766394352.629121 mapwrite 0x3e6631 thru 0x3e8570 (0x1f40 bytes) 000071[1] 1766394352.650394 mapread 0x73e89c thru 0x73efb5 (0x71a bytes) 000072[1] 1766394352.657774 read 0x93cf67 thru 0x93cfe7 (0x81 bytes) 000074[1] 1766394352.666187 trunc from 0x9c0064 to 0x0afeef (0x910174 bytes) 000076[0] 1766394352.680776 mapread 0x0ab031 thru 0x0ab737 (0x707 bytes) 000077[0] 1766394352.687567 read 0x053b6a thru 0x0544e5 (0x97c bytes) 000081[0] 1766394352.701185 trunc from 0x0afef0 to 0x2864ee (0x1d65ff bytes) 000084[0] 1766394352.719311 mapread 0x23f143 thru 0x24042f (0x12ed bytes) 000086[0] 1766394352.737887 mapwrite 0x64746c thru 0x6480de (0xc73 bytes) 000090[1] 1766394352.783393 read 0x549429 thru 0x549d01 (0x8d9 bytes) 000091[1] 1766394352.792304 mapread 0x467ff6 thru 0x4680d2 (0xdd bytes) 000092[1] 1766394352.797659 trunc from 0x6480df to 0x4a58e7 (0x1a27f7 bytes) 000094[1] 1766394352.809411 trunc from 0x4a58e8 to 0x724409 (0x27eb22 bytes) 000095[1] 1766394352.813518 mapread 0x0b43de thru 0x0b5f10 (0x1b33 bytes) 000102[0] 1766394352.826167 mapread 0x66aa1a thru 0x66c666 (0x1c4d bytes) 000103[1] 1766394352.839949 write 0x771bea thru 0x773573 (0x198a bytes) 000105[1] 1766394352.846957 mapwrite 0x473850 thru 0x47409b (0x84c bytes) 000106[1] 1766394352.863788 trunc from 0x773574 to 0x10ee69 (0x66470a bytes) 000107[0] 1766394352.872661 read 0x04c2ca thru 0x04de38 (0x1b6f bytes) 000109[0] 1766394352.887856 write 0x431210 thru 0x431636 (0x427 bytes) 000112[0] 1766394352.894927 mapwrite 0x89c84c thru 0x89cc76 (0x42b bytes) 000113[0] 1766394352.939366 mapwrite 0x934cd6 thru 0x9362a5 (0x15d0 bytes) 000115[0] 1766394352.985610 read 0x264485 thru 0x2645cf (0x14b bytes) 000118[1] 1766394352.993220 write 0x2788d9 thru 0x27a2d0 (0x19f8 bytes) 000119[0] 1766394353.004231 trunc from 0x9362a6 to 0x29813b (0x69e16a bytes) 000120[0] 1766394353.018006 trunc from 0x29813c to 0x96fa03 (0x6d78c8 bytes) 000121[0] 1766394353.026976 mapwrite 0x376f6c thru 0x378dc0 (0x1e55 bytes) 000122[0] 1766394353.045438 trunc from 0x96fa04 to 0x532332 (0x43d6d1 bytes) 000124[1] 1766394353.057924 read 0x438126 thru 0x43935c (0x1237 bytes) 000125[0] 1766394353.063478 mapwrite 0x16ce2e thru 0x16ea1d (0x1bf0 bytes) 000127[1] 1766394353.083787 mapread 0x507e16 thru 0x5098bb (0x1aa6 bytes) 000132[1] 1766394353.102390 trunc from 0x532333 to 0x596cec (0x649ba bytes) 000133[1] 1766394353.110937 write 0x132a84 thru 0x132d60 (0x2dd bytes) 000134[0] 1766394353.112540 mapwrite 0x7524ac thru 0x753ce1 (0x1836 bytes) 000135[1] 1766394353.138410 write 0x202b76 thru 0x204015 (0x14a0 bytes) 000136[0] 1766394353.146499 mapread 0x083a86 thru 0x083aff (0x7a bytes) 000137[0] 1766394353.156021 mapread 0x601a85 thru 0x602bbd (0x1139 bytes) 000139[1] 1766394353.163431 trunc from 0x753ce2 to 0x100650 (0x653691 bytes) 000140[1] 1766394353.176457 mapwrite 0x7f4ec5 thru 0x7f57cf (0x90b bytes) 000142[0] 1766394353.199949 read 0x45a4bf thru 0x45a6fe (0x240 bytes) 000143[0] 1766394353.206394 mapwrite 0x41f59c thru 0x420a32 (0x1497 bytes) 000147[0] 1766394353.236061 mapwrite 0x4225ca thru 0x4231c4 (0xbfb bytes) 000148[1] 1766394353.253489 mapread 0x7d28bf thru 0x7d30da (0x81c bytes) 000149[1] 1766394353.266281 mapwrite 0x2dae3e thru 0x2dc258 (0x141b bytes) 000151[0] 1766394353.285476 read 0x664a54 thru 0x66680d (0x1dba bytes) 000154[1] 1766394353.290598 read 0x73ca21 thru 0x73e8ed (0x1ecd bytes) 000157[1] 1766394353.301178 read 0x27a478 thru 0x27b165 (0xcee bytes) 000159[0] 1766394353.309388 mapread 0x645b1e thru 0x64617a (0x65d bytes) 000160[0] 1766394353.316405 mapread 0x39f92d thru 0x3a0db9 (0x148d bytes) 000161[0] 1766394353.322401 trunc from 0x7f57d0 to 0x8ec02d (0xf685e bytes) 000162[1] 1766394353.335424 mapwrite 0x154d62 thru 0x155c28 (0xec7 bytes) 000163[0] 1766394353.354901 mapread 0x3718f5 thru 0x372620 (0xd2c bytes) 000164[0] 1766394353.362172 trunc from 0x8ec02e to 0x79584e (0x1567df bytes) 000165[0] 1766394353.368541 mapread 0x51bbd5 thru 0x51c769 (0xb95 bytes) 000166[0] 1766394353.371170 mapwrite 0x84f08d thru 0x8502e4 (0x1258 bytes) 000167[0] 1766394353.387187 trunc from 0x8502e5 to 0x1a2d1b (0x6ad5c9 bytes) 000170[1] 1766394353.396258 trunc from 0x1a2d1c to 0x41ec30 (0x27bf15 bytes) 000175[0] 1766394353.419798 write 0x400643 thru 0x401142 (0xb00 bytes) 000176[1] 1766394353.426919 mapwrite 0x2e3691 thru 0x2e4b02 (0x1472 bytes) 000177[1] 1766394353.444114 trunc from 0x41ec31 to 0x771fed (0x3533bd bytes) 000179[1] 1766394353.457699 trunc from 0x771fee to 0x27ef59 (0x4f3094 bytes) 000181[1] 1766394353.466663 read 0x238896 thru 0x238ef2 (0x65d bytes) 000183[0] 1766394353.486019 trunc from 0x27ef5a to 0x198f9f (0xe5fba bytes) 000185[0] 1766394353.496784 mapread 0x05e46e thru 0x06039a (0x1f2d bytes) 000187[0] 1766394353.506192 read 0x173267 thru 0x173b36 (0x8d0 bytes) 000188[0] 1766394353.516385 read 0x12d059 thru 0x12e86d (0x1815 bytes) 000193[0] 1766394353.524456 write 0x4068c6 thru 0x406eac (0x5e7 bytes) 000195[0] 1766394353.529387 read 0x39c1e6 thru 0x39cca8 (0xac3 bytes) 000196[0] 1766394353.534307 read 0x0f10c3 thru 0x0f2d11 (0x1c4f bytes) 000197[1] 1766394353.537124 mapread 0x188802 thru 0x18986d (0x106c bytes) 000200[0] 1766394353.549754 mapwrite 0x00da8e thru 0x00f795 (0x1d08 bytes) 000201[1] 1766394353.564249 mapread 0x39cb1f thru 0x39d3e1 (0x8c3 bytes) 000202[1] 1766394353.577491 mapwrite 0x474b74 thru 0x4757b9 (0xc46 bytes) 000203[0] 1766394353.611677 read 0x08f846 thru 0x090935 (0x10f0 bytes) 000204[1] 1766394353.616975 mapwrite 0x1233ec thru 0x124c4f (0x1864 bytes) 000206[0] 1766394353.639436 mapread 0x14dc79 thru 0x14ef13 (0x129b bytes) 000207[1] 1766394353.646452 mapwrite 0x13da00 thru 0x13f39b (0x199c bytes) 000209[1] 1766394353.665369 mapread 0x3258e5 thru 0x326a69 (0x1185 bytes) 000210[0] 1766394353.668343 write 0x6d7081 thru 0x6d8fe5 (0x1f65 bytes) 000212[0] 1766394353.678480 mapread 0x5a3456 thru 0x5a4340 (0xeeb bytes) 000213[0] 1766394353.685450 write 0x933bda thru 0x9358a2 (0x1cc9 bytes) 000214[1] 1766394353.688904 mapread 0x6a3d5e thru 0x6a43b6 (0x659 bytes) 000215[1] 1766394353.710927 mapread 0x55b7a5 thru 0x55cb39 (0x1395 bytes) 000217[1] 1766394353.716170 mapwrite 0x87776b thru 0x877890 (0x126 bytes) 000226[1] 1766394353.746569 trunc from 0x9358a3 to 0x06ba51 (0x8c9e51 bytes) 000231[0] 1766394353.767477 trunc from 0x06ba52 to 0x07211c (0x66cb bytes) 000232[0] 1766394353.779472 read 0x02be75 thru 0x02ced2 (0x105e bytes) 000234[1] 1766394353.789880 trunc from 0x07211d to 0x4a0a18 (0x42e8fc bytes) 000237[1] 1766394353.809568 trunc from 0x4a0a19 to 0x42e0d0 (0x72948 bytes) 000239[0] 1766394353.822212 trunc from 0x42e0d1 to 0x87bb55 (0x44da85 bytes) 000242[1] 1766394353.836138 trunc from 0x87bb56 to 0x69512b (0x1e6a2a bytes) 000244[0] 1766394353.849746 write 0x0e2ac7 thru 0x0e4678 (0x1bb2 bytes) 000245[1] 1766394353.879239 mapread 0x104f54 thru 0x1066a6 (0x1753 bytes) 000246[1] 1766394353.890747 write 0x50bc8c thru 0x50d3d9 (0x174e bytes) 000249[1] 1766394353.897441 trunc from 0x69512c to 0x16aa9d (0x52a68e bytes) 000250[0] 1766394353.908144 write 0x10de64 thru 0x10f06b (0x1208 bytes) 000253[0] 1766394353.924253 mapread 0x0714fe thru 0x072e53 (0x1956 bytes) 000254[1] 1766394353.931570 trunc from 0x16aa9e to 0x993954 (0x828eb7 bytes) 000257[1] 1766394353.953539 mapwrite 0x6de9ee thru 0x6dee5b (0x46e bytes) 000258[0] 1766394353.976616 mapread 0x158980 thru 0x159730 (0xdb1 bytes) 000260[1] 1766394353.985213 trunc from 0x993955 to 0x256f35 (0x73ca1f bytes) 000262[1] 1766394353.999053 trunc from 0x256f36 to 0x683f0c (0x42cfd7 bytes) 000263[1] 1766394354.008571 write 0x07d2de thru 0x07dd69 (0xa8c bytes) 000264[1] 1766394354.011547 trunc from 0x683f0d to 0x3fff7f (0x283f8d bytes) 000265[0] 1766394354.016045 read 0x0923ba thru 0x0927a0 (0x3e7 bytes) 000266[1] 1766394354.023970 mapread 0x2df0ee thru 0x2e100c (0x1f1f bytes) 000270[0] 1766394354.032629 write 0x43cc25 thru 0x43da89 (0xe65 bytes) 000271[1] 1766394354.035226 read 0x2584b0 thru 0x25a18b (0x1cdc bytes) 000273[0] 1766394354.063319 mapread 0x2f43bc thru 0x2f5c95 (0x18da bytes) 000274[0] 1766394354.066055 mapread 0x1bd569 thru 0x1bf194 (0x1c2c bytes) 000277[0] 1766394354.077464 trunc from 0x43da8a to 0x095d99 (0x3a7cf0 bytes) 000279[0] 1766394354.093920 trunc from 0x095d9a to 0x680ae5 (0x5ead4c bytes) 000280[0] 1766394354.104232 trunc from 0x680ae6 to 0x9a5305 (0x324820 bytes) 000281[0] 1766394354.113180 trunc from 0x9a5306 to 0x8fa04a (0xab2bb bytes) 000284[1] 1766394354.123049 read 0x5e75b0 thru 0x5e917e (0x1bcf bytes) 000286[1] 1766394354.130612 read 0x7bf392 thru 0x7bfa4d (0x6bc bytes) 000290[0] 1766394354.141001 trunc from 0x8fa04b to 0x4e6901 (0x413749 bytes) 000291[0] 1766394354.147735 mapwrite 0x10067a thru 0x102168 (0x1aef bytes) 000292[0] 1766394354.159827 trunc from 0x4e6902 to 0x150b2f (0x395dd2 bytes) 000293[0] 1766394354.167453 mapread 0x042d2e thru 0x044382 (0x1655 bytes) 000294[1] 1766394354.177085 read 0x076bd4 thru 0x077d10 (0x113d bytes) 000295[0] 1766394354.192509 mapwrite 0x41a4df thru 0x41bf1d (0x1a3f bytes) 000301[0] 1766394354.231369 read 0x17e933 thru 0x17f0b7 (0x785 bytes) 000302[0] 1766394354.237199 read 0x30bf61 thru 0x30c29a (0x33a bytes) 000303[1] 1766394354.238950 trunc from 0x41bf1e to 0x717ad4 (0x2fbbb7 bytes) 000304[1] 1766394354.249349 write 0x57b3bc thru 0x57cecf (0x1b14 bytes) 000308[1] 1766394354.277278 write 0x37e753 thru 0x37e828 (0xd6 bytes) 000309[1] 1766394354.283725 mapwrite 0x3f1044 thru 0x3f13cd (0x38a bytes) 000312[1] 1766394354.299539 mapwrite 0x1d1365 thru 0x1d206c (0xd08 bytes) 000315[1] 1766394354.314577 trunc from 0x717ad5 to 0x4c61c2 (0x251912 bytes) 000318[1] 1766394354.328048 mapwrite 0x5a1756 thru 0x5a35dd (0x1e88 bytes) 000319[0] 1766394354.354383 write 0x930c86 thru 0x930d10 (0x8b bytes) 000320[1] 1766394354.366164 trunc from 0x930d11 to 0x671af7 (0x2bf219 bytes) 000321[1] 1766394354.374949 mapread 0x3d5fc3 thru 0x3d7562 (0x15a0 bytes) 000324[0] 1766394354.383337 write 0x282bbb thru 0x2844d8 (0x191e bytes) 000325[0] 1766394354.390712 mapread 0x0950ab thru 0x09610e (0x1064 bytes) 000326[0] 1766394354.397971 write 0x9d14c6 thru 0x9d17c4 (0x2ff bytes) 000327[0] 1766394354.402153 mapread 0x12cfb0 thru 0x12d3a2 (0x3f3 bytes) 000328[0] 1766394354.409972 trunc from 0x9d17c5 to 0x5d4e14 (0x3fc9b0 bytes) 000329[1] 1766394354.418932 mapread 0x3a03d9 thru 0x3a110e (0xd36 bytes) 000330[0] 1766394354.432164 mapwrite 0x722d7f thru 0x7232eb (0x56d bytes) 000331[1] 1766394354.452869 mapwrite 0x0f6775 thru 0x0f6eba (0x746 bytes) 000332[1] 1766394354.472915 mapwrite 0x067935 thru 0x0681d1 (0x89d bytes) 000336[0] 1766394354.499572 read 0x276979 thru 0x2786ae (0x1d36 bytes) 000341[0] 1766394354.507545 read 0x57a32c thru 0x57ab16 (0x7eb bytes) 000342[0] 1766394354.509143 write 0x475098 thru 0x47513e (0xa7 bytes) 000343[1] 1766394354.510607 trunc from 0x7232ec to 0x453aa5 (0x2cf846 bytes) 000346[1] 1766394354.525486 trunc from 0x453aa6 to 0x5fa7d7 (0x1a6d32 bytes) 000347[0] 1766394354.530308 write 0x8a33e6 thru 0x8a424a (0xe65 bytes) 000349[1] 1766394354.541836 trunc from 0x8a424b to 0x120c07 (0x783643 bytes) truncating to largest ever: 0x9f224d 000354[0] 1766394354.564243 trunc from 0x120c08 to 0x9f224c (0x8d1645 bytes) 000356[1] 1766394354.575085 read 0x91796f thru 0x918682 (0xd14 bytes) 000360[0] 1766394354.603475 read 0x22dbf5 thru 0x22e50e (0x91a bytes) 000362[0] 1766394354.612208 trunc from 0x9f224d to 0x8566ed (0x19bb5f bytes) 000367[0] 1766394354.635194 trunc from 0x8566ee to 0x3f21e4 (0x464509 bytes) 000368[0] 1766394354.643566 mapwrite 0x9d9285 thru 0x9daf8e (0x1d0a bytes) 000369[1] 1766394354.666159 read 0x392b11 thru 0x394977 (0x1e67 bytes) 000371[1] 1766394354.675363 mapwrite 0x7acd76 thru 0x7ade69 (0x10f4 bytes) 000372[0] 1766394354.693310 read 0x91f128 thru 0x91fa01 (0x8da bytes) 000373[1] 1766394354.695296 write 0x7ece96 thru 0x7ee42b (0x1596 bytes) 000374[0] 1766394354.701729 trunc from 0x9daf8f to 0x5c520b (0x415d83 bytes) 000375[0] 1766394354.716599 mapread 0x569b58 thru 0x569d5c (0x205 bytes) 000377[0] 1766394354.723673 mapread 0x4489d7 thru 0x448f40 (0x56a bytes) 000378[1] 1766394354.725697 mapwrite 0x7af7b0 thru 0x7afce3 (0x534 bytes) 000379[0] 1766394354.748492 write 0x32e391 thru 0x32ecdd (0x94d bytes) 000380[1] 1766394354.756933 mapread 0x004d38 thru 0x00614d (0x1416 bytes) 000381[0] 1766394354.771337 trunc from 0x7afce4 to 0x8a043b (0xf0758 bytes) 000382[1] 1766394354.780817 mapwrite 0x101510 thru 0x102df0 (0x18e1 bytes) 000385[0] 1766394354.810686 read 0x350ab8 thru 0x3521dd (0x1726 bytes) 000388[0] 1766394354.818561 read 0x07a370 thru 0x07ab2e (0x7bf bytes) 000394[0] 1766394354.833210 mapwrite 0x2f199b thru 0x2f2544 (0xbaa bytes) 000395[1] 1766394354.852233 read 0x767fcc thru 0x76961c (0x1651 bytes) 000398[1] 1766394354.863924 mapwrite 0x672af5 thru 0x672dbe (0x2ca bytes) 000400[1] 1766394354.886546 read 0x070e74 thru 0x07235b (0x14e8 bytes) 000401[1] 1766394354.893180 mapwrite 0x934fdb thru 0x93665d (0x1683 bytes) 000402[0] 1766394354.914784 write 0x906a0c thru 0x90808f (0x1684 bytes) 000403[0] 1766394354.924223 mapwrite 0x4bc719 thru 0x4bce95 (0x77d bytes) 000404[0] 1766394354.941425 write 0x5dda53 thru 0x5de4f6 (0xaa4 bytes) 000405[1] 1766394354.949207 write 0x147807 thru 0x148845 (0x103f bytes) 000406[1] 1766394354.959488 read 0x4b8531 thru 0x4b9647 (0x1117 bytes) 000407[1] 1766394354.964502 mapwrite 0x65bc52 thru 0x65c0ac (0x45b bytes) 000412[1] 1766394354.984337 mapread 0x104493 thru 0x10575c (0x12ca bytes) 000414[1] 1766394354.990750 mapread 0x609d15 thru 0x60a3b1 (0x69d bytes) 000416[1] 1766394354.996798 write 0x0571c3 thru 0x0577a1 (0x5df bytes) 000419[0] 1766394355.002839 read 0x592969 thru 0x592b73 (0x20b bytes) 000421[1] 1766394355.007701 mapread 0x599089 thru 0x599c5e (0xbd6 bytes) 000425[0] 1766394355.016746 read 0x1d4a90 thru 0x1d506f (0x5e0 bytes) 000426[0] 1766394355.023285 trunc from 0x93665e to 0x427d02 (0x50e95b bytes) 000427[1] 1766394355.033405 read 0x154915 thru 0x154d99 (0x485 bytes) 000428[0] 1766394355.042774 write 0x3b9101 thru 0x3baa53 (0x1953 bytes) 000430[0] 1766394355.057260 read 0x3639d5 thru 0x3653e4 (0x1a10 bytes) 000432[0] 1766394355.063351 trunc from 0x427d03 to 0x54a486 (0x122784 bytes) 000438[0] 1766394355.086335 trunc from 0x54a487 to 0x390ed5 (0x1b95b1 bytes) 000440[0] 1766394355.095659 mapread 0x3467a1 thru 0x3486fd (0x1f5d bytes) 000443[0] 1766394355.099460 trunc from 0x390ed6 to 0x383b78 (0xd35d bytes) 000445[0] 1766394355.111540 mapwrite 0x9aa08d thru 0x9ab0ca (0x103e bytes) 000448[0] 1766394355.141967 mapwrite 0x057364 thru 0x0574a0 (0x13d bytes) 000449[1] 1766394355.153903 trunc from 0x9ab0cb to 0x6547bc (0x35690e bytes) 000450[1] 1766394355.167411 read 0x449903 thru 0x44b109 (0x1807 bytes) 000451[1] 1766394355.172758 write 0x23007f thru 0x231edf (0x1e61 bytes) 000453[0] 1766394355.176328 write 0x65be09 thru 0x65be63 (0x5b bytes) 000457[1] 1766394355.192122 read 0x54f24e thru 0x54fedb (0xc8e bytes) 000460[1] 1766394355.209927 read 0x012b60 thru 0x013169 (0x60a bytes) 000461[1] 1766394355.211581 trunc from 0x65be64 to 0x7b956e (0x15d70b bytes) 000462[0] 1766394355.222614 read 0x2f1ef1 thru 0x2f3e91 (0x1fa1 bytes) 000463[0] 1766394355.229075 trunc from 0x7b956f to 0x6ccc2d (0xec941 bytes) 000465[1] 1766394355.246968 read 0x305e9f thru 0x3074dd (0x163f bytes) 000466[0] 1766394355.252837 read 0x032b92 thru 0x0340c9 (0x1538 bytes) 000468[0] 1766394355.258174 mapwrite 0x383c68 thru 0x3847ed (0xb86 bytes) 000469[1] 1766394355.273175 mapread 0x088a88 thru 0x089830 (0xda9 bytes) 000473[0] 1766394355.287892 trunc from 0x6ccc2e to 0x5be54e (0x10e6df bytes) 000474[1] 1766394355.299502 mapread 0x03eee2 thru 0x03ff76 (0x1095 bytes) 000475[0] 1766394355.309483 mapwrite 0x38645a thru 0x387ec3 (0x1a6a bytes) 000477[1] 1766394355.330628 trunc from 0x5be54f to 0x33612e (0x288420 bytes) 000479[0] 1766394355.344559 mapread 0x33361a thru 0x33519e (0x1b85 bytes) 000482[0] 1766394355.355946 read 0x2ac684 thru 0x2ae5f8 (0x1f75 bytes) 000483[1] 1766394355.380048 read 0x1b98fb thru 0x1bad22 (0x1428 bytes) 000484[1] 1766394355.403413 write 0x8fdecd thru 0x8fe11e (0x252 bytes) 000486[1] 1766394355.415531 mapwrite 0x0ebe60 thru 0x0ec13b (0x2dc bytes) 000487[1] 1766394355.438635 mapwrite 0x118424 thru 0x119eef (0x1acc bytes) 000490[1] 1766394355.463725 mapread 0x7baae2 thru 0x7bca81 (0x1fa0 bytes) 000492[0] 1766394355.491308 mapwrite 0x914602 thru 0x91494f (0x34e bytes) 000494[1] 1766394355.527970 mapread 0x25f15d thru 0x260eb2 (0x1d56 bytes) 000496[1] 1766394355.539401 mapwrite 0x32577b thru 0x327235 (0x1abb bytes) 000499[0] 1766394355.567118 read 0x78ef38 thru 0x790b38 (0x1c01 bytes) 000500[1] 1766394355.572315 mapwrite 0x697d73 thru 0x699a7c (0x1d0a bytes) All operations completed A-OK! PASS 16b (6s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:05:57 (1766394357) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:05:58 (1766394358) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.085814 s, 97.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0610302 s, 137 MB/s ldlm.namespaces.MGC192.168.204.139@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff93f847fb8000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff93f8483b5000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff93f8483b5000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff93f8483b5000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.46999 s, 42.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.640747 s, 164 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.766584 s, 137 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.38717 s, 75.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.54301 s, 68.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.0559 s, 99.3 MB/s PASS 16d (11s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:06:09 (1766394369) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0875925 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352705 s, 29.7 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:06:11 (1766394371) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:06:34 (1766394394) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 04:06:56 (1766394416) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 04:06:58 (1766394418) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 04:07:00 (1766394420) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.489442 s, 53.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.331858 s, 79.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.437113 s, 60.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.307881 s, 85.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.420201 s, 62.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.421621 s, 62.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.525824 s, 49.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.560534 s, 46.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.577015 s, 45.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.589249 s, 44.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.596012 s, 44.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.422228 s, 62.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.458981 s, 57.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.559855 s, 46.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.589795 s, 44.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.559354 s, 46.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.645912 s, 40.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.692127 s, 37.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.738741 s, 35.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (8s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:07:08 (1766394428) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x32cce75 fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x3c569db truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Dec 22 04:07:19 EST 2025 + fsx 2 done: rc=0 Mon Dec 22 04:07:19 EST 2025 + fsx 3 done: rc=0 Mon Dec 22 04:07:19 EST 2025 + fsx 4 done: rc=0 Mon Dec 22 04:07:19 EST 2025 + fsx 5 done: rc=0 Mon Dec 22 04:07:19 EST 2025 PASS 16k (13s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:07:22 (1766394442) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 04:07:26 (1766394446) mmap test1: basic mmap operation (PASS, 0.029379s) mmap test2: MAP_PRIVATE not write back (PASS, 0.017847s) mmap test3: concurrent mmap ops on two nodes (PASS, 11.004s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.089s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.036409s) mmap test6: check mmap write/read content on two nodes (PASS, 0.042214s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.42283s) mmap test11: multi-thread shared mmap (PASS, 0.41633s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0186s) PASS 18 (20s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:07:46 (1766394466) oleg439-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:07:48 (1766394468) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:07:50 (1766394470) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 04:07:52 (1766394472) now is 1766394473 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 new atime is 1766394534 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:08:56 (1766394536) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234852 388 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre[OST:1] filesystem_summary: 234852 388 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117198 398 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234852 388 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:08:59 (1766394539) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:09:01 (1766394541) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:09:03 (1766394543) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 04:09:05 (1766394545) utime: good mknod times 1766394544 <= 1766394545 <= 1766394545 for /mnt/lustre/f26a utime: good mknod times 1766394544 <= 1766394545 <= 1766394545 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 04:09:08 (1766394548) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 04:09:12 (1766394552) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 04:09:13.000000000 -0500 Birth: 2025-12-22 04:09:13.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 04:09:13.000000000 -0500 Birth: 2025-12-22 04:09:13.000000000 -0500 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 04:09:16 (1766394556) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:09:21 (1766394561) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:09:24 (1766394564) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg439-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.139@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 04:09:43 (1766394583) -rw-r--r-- 1 root root 0 Dec 22 04:09 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 04:09:48 (1766394588) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42950 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 04:09:51 (1766394591) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:09:55 (1766394595) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 04:09:56 (1766394596) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 04:09:57 (1766394597) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 04:09:58 (1766394598) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 04:09:59 (1766394599) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid in FULL state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid in IDLE state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid in FULL state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid in IDLE state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f847fb8000.ost_server_uuid in IDLE state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93f847fb8000.ost_server_uuid in IDLE state after 0 sec PASS 34 (92s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 04:11:31 (1766394691) fail_loc=0x80000317 Race attempt 0 Wait for 49070 49204 for 60 sec... total: 4000 open/close in 8.49 seconds: 471.05 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.139@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.139@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 04:12:36 (1766394756) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg439-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.354098 s, 148 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.329437 s, 159 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.489637 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.412663 s, 127 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.450108 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.460464 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.408887 s, 128 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.393705 s, 133 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.481477 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.408258 s, 128 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.385293 s, 136 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (231s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 04:16:27 (1766394987) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 - create 8759 (time 1766394998.73 total 10.00 last 875.82) total: 10000 create in 11.42 seconds: 875.86 ops/second PASS 37 (15s) == sanityn test 39a: file mtime does not change after rename ========================================================== 04:16:42 (1766395002) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 04:16:45 (1766395005) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 04:16:48 (1766395008) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 04:16:52 (1766395012) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 04:16:54 (1766395014) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 04:17:03 (1766395023) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 04:17:11 (1766395031) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 04:17:19 (1766395039) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 04:17:27 (1766395047) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 04:17:32 (1766395052) open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 04:17:38 (1766395058) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 04:17:44 (1766395064) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 04:17:49 (1766395069) Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 04:17:55 (1766395075) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 04:18:00 (1766395080) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 04:18:05 (1766395085) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 04:18:11 (1766395091) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (725s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 04:30:16 (1766395816) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 04:30:21 (1766395821) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 04:30:27 (1766395827) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 04:30:33 (1766395833) Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 04:30:38 (1766395838) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 04:30:43 (1766395843) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 04:30:49 (1766395849) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 04:30:54 (1766395854) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 04:31:00 (1766395860) PASS 43a (15s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 04:31:15 (1766395875) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 04:31:21 (1766395881) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 04:31:26 (1766395886) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 04:31:31 (1766395891) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 04:31:37 (1766395897) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 04:31:42 (1766395902) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 04:31:48 (1766395908) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 04:31:53 (1766395913) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 04:31:54 (1766395914) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (46s) == sanityn test 43k: unlink vs create ==================== 04:32:40 (1766395960) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (615s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 04:42:55 (1766396575) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Dec 22 04:43:01 EST 2025 PASS 44a (10s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 04:43:05 (1766396585) open: File exists Conflict PASS 44b (9s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 04:43:14 (1766396594) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (9s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 04:43:24 (1766396604) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 04:43:34 (1766396614) Conflict PASS 44e (10s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 04:43:44 (1766396624) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 04:43:53 (1766396633) Conflict PASS 44g (9s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 04:44:02 (1766396642) Conflict PASS 44h (11s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 04:44:13 (1766396653) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 04:44:15 (1766396655) PASS 45a (101s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 04:45:56 (1766396756) Conflict PASS 45b (11s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 04:46:08 (1766396768) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 04:46:20 (1766396780) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (10s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 04:46:30 (1766396790) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 04:46:40 (1766396800) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (9s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 04:46:49 (1766396809) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 04:46:59 (1766396819) Conflict PASS 45h (10s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 04:47:09 (1766396829) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 04:47:12 (1766396832) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (802s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 05:00:34 (1766397634) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 05:00:41 (1766397641) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 05:00:47 (1766397647) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 05:00:54 (1766397654) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 05:01:01 (1766397661) Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 05:01:07 (1766397667) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 05:01:14 (1766397674) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 05:01:20 (1766397680) Conflict PASS 46h (7s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 05:01:27 (1766397687) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 05:01:28 (1766397688) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 05:01:30 (1766397690) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 05:01:31 (1766397691) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 05:01:33 (1766397693) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 05:01:34 (1766397694) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 05:01:35 (1766397695) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 05:01:37 (1766397697) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 05:01:38 (1766397698) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0131314 s, 780 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 05:01:46 (1766397706) 0+1 records in 0+1 records out 158 bytes copied, 0.00690166 s, 22.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260455) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 05:01:51 (1766397711) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00502844 s, 204 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038474 s, 27.3 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 05:02:06 (1766397726) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00535818 s, 191 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 05:02:13 (1766397733) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176388 s, 59.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 05:02:17 (1766397737) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 05:02:21 (1766397741) llite.lustre-ffff93f847fb8000.xattr_cache=1 llite.lustre-ffff93f8483b5000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:22.000000000 -0500 Modify: 2025-12-22 05:02:22.000000000 -0500 Change: 2025-12-22 05:02:22.000000000 -0500 Birth: 2025-12-22 05:02:22.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:27.000000000 -0500 Modify: 2025-12-22 05:02:28.000000000 -0500 Change: 2025-12-22 05:02:28.000000000 -0500 Birth: 2025-12-22 05:02:28.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:33.000000000 -0500 Modify: 2025-12-22 05:02:33.000000000 -0500 Change: 2025-12-22 05:02:33.000000000 -0500 Birth: 2025-12-22 05:02:33.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:33.000000000 -0500 Modify: 2025-12-22 05:02:33.000000000 -0500 Change: 2025-12-22 05:02:33.000000000 -0500 Birth: 2025-12-22 05:02:33.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:38.000000000 -0500 Modify: 2025-12-22 05:02:39.000000000 -0500 Change: 2025-12-22 05:02:39.000000000 -0500 Birth: 2025-12-22 05:02:39.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:02:38.000000000 -0500 Modify: 2025-12-22 05:02:39.000000000 -0500 Change: 2025-12-22 05:02:39.000000000 -0500 Birth: 2025-12-22 05:02:39.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 05:02:46 (1766397766) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 05:02:54 (1766397774) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 05:03:02 (1766397782) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 05:03:15 (1766397795) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 05:03:25 (1766397805) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 05:03:26 (1766397806) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 05:03:39 (1766397819) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 05:03:54 (1766397834) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1766397834 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.74397 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (37s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 05:04:31 (1766397871) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1766397872 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.21248 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (42s) == sanityn test 60: Verify data_version behaviour ======== 05:05:13 (1766397913) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 05:05:16 (1766397916) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 05:05:19 (1766397919) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 05:05:22 (1766397922) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0041194 s, 15.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 05:05:23 (1766397923) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00625414 s, 10.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 05:05:25 (1766397925) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 05:05:26 (1766397926) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00945757 s, 6.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 05:05:28 (1766397928) llite.lustre-ffff93f847fb8000.xattr_cache=1 llite.lustre-ffff93f8483b5000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 05:05:30 (1766397930) llite.lustre-ffff93f847fb8000.xattr_cache=1 llite.lustre-ffff93f8483b5000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 05:05:33 (1766397933) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00567156 s, 181 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00453541 s, 226 kB/s 273215: get lock2 273217: get lock1 273217: done 273217: get lock2 273215: done 273215: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 273215: cannot get lock1: Resource deadlock avoided 273217: done 273217: exit rc=0 273215: done 273215: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 05:05:39 (1766397939) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0245644 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143386 s, 73.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273798 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 05:05:47 (1766397947) opening files: ........ closing files: ........ PASS 76 (23s) == sanityn test 77a: check FIFO NRS policy =============== 05:06:10 (1766397970) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141961 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685674 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957995 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877222 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10754 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102366 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148298 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140582 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132807 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172206 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160233 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15358 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170592 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154823 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171386 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162499 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183258 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262615 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025528 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020111 s, 52.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254447 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284012 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228534 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286911 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187667 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975338 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250417 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265767 s, 39.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189739 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731878 s, 143 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815929 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141432 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719804 s, 146 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 05:06:13 (1766397973) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159857 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222692 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368639 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387246 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538988 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518404 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617026 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834332 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676324 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619852 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775696 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848511 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108401 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982648 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926045 s, 11.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100301 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942573 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161768 s, 64.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206663 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227726 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125389 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131064 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187233 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184912 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021292 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202615 s, 51.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481807 s, 218 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207854 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779372 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165609 s, 63.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958296 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032038 s, 327 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430456 s, 244 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12887 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297415 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357941 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431582 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553566 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453025 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642504 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626841 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636957 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536836 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750128 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902116 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081754 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890749 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991933 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105012 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103838 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148628 s, 70.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174388 s, 60.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102123 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130448 s, 80.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010992 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219063 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339681 s, 309 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397553 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411027 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385192 s, 272 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164566 s, 63.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00730664 s, 144 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022529 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408306 s, 257 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436744 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777609 s, 135 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 05:06:17 (1766397977) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168833 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350801 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400494 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023126 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766263 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726116 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112235 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100134 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121311 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114695 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10039 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110856 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116082 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108641 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121931 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120027 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130481 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192385 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137548 s, 76.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269357 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014861 s, 70.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211175 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228841 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156614 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235065 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256811 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025339 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287011 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265917 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276847 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308854 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279155 s, 37.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585626 s, 17.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174945 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341297 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921686 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991656 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108911 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900969 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991628 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116967 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906883 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124482 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112675 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115212 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140734 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170436 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209392 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483886 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403563 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283222 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404172 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309129 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232544 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201044 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511912 s, 20.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330689 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458872 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424139 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629615 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399962 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625967 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521986 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467837 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210665 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564452 s, 18.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149862 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195333 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402892 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943764 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971144 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828641 s, 12.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1252 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129111 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118402 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115185 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117469 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125383 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132553 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155083 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146126 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116277 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151327 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232213 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765532 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112962 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233781 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283748 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136343 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241841 s, 43.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201808 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362084 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171497 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767501 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161696 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462962 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155854 s, 67.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112463 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180049 s, 58.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 05:06:23 (1766397983) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.15519 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164946 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659112 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697403 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716862 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837771 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637595 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707524 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838328 s, 12.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102622 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357114 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438469 s, 23.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352449 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418944 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327182 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187576 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139468 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194678 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258384 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182538 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188073 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024612 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427869 s, 24.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326074 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172568 s, 60.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349656 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433636 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297784 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324612 s, 32.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199159 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197536 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289019 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189628 s, 5.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135991 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210741 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630451 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517266 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656611 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939345 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938207 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972606 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925847 s, 11.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822225 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892025 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826368 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109535 s, 9.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099853 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928121 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962519 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100916 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180794 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263834 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255059 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393217 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333975 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140394 s, 74.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329789 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255302 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163393 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262533 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334602 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174733 s, 60.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246584 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122576 s, 85.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024606 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551473 s, 190 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141279 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317541 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460162 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588391 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529536 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602017 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488993 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664319 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393974 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690385 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120197 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120522 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107449 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119463 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118815 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127997 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153745 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127305 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231757 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274685 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024043 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370552 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253133 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282786 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337067 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350852 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323361 s, 32.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408346 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323031 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357408 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565457 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408472 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408049 s, 25.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 05:06:28 (1766397988) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.39@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.187546 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220777 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333555 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085807 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101462 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885195 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109991 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881002 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109286 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116728 s, 9.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962064 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848191 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113277 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104497 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10557 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111187 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917935 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173605 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166142 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179979 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166663 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172446 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180857 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182643 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180816 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182321 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179944 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182748 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181614 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179281 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180694 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185745 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185956 s, 5.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139668 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529633 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043039 s, 24.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617059 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871849 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113444 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15496 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157585 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15792 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164943 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16893 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158121 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180978 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162476 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168627 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153742 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171433 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928964 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993732 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933111 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109337 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953082 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104384 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106992 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100895 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110362 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114691 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115073 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103677 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101469 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107271 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105267 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11558 s, 9.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153477 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241616 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630702 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589664 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061885 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580297 s, 18.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660564 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730323 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481163 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673557 s, 15.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944425 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097504 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986394 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10975 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083637 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114065 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119986 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176571 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182958 s, 57.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259243 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123914 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102722 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242264 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239172 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205458 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216853 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288184 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382423 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278727 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263479 s, 39.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021581 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005657 s, 185 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100419 s, 104 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131661 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226811 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256616 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165006 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178427 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184027 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192595 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199031 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189174 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192133 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208341 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201549 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200635 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197459 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210143 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199935 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237002 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024959 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280972 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273475 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374653 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285829 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372234 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389341 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291145 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00819231 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412782 s, 25.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346508 s, 30.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435609 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216844 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656852 s, 16.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710307 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319234 s, 328 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 05:06:36 (1766397996) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164889 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325377 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983927 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119454 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137057 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130199 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118426 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150988 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132752 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125699 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139918 s, 7.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151171 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146645 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133224 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16166 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1401 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169372 s, 61.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236051 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214052 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021458 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255482 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220531 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129587 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243926 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302577 s, 34.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029658 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954765 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913859 s, 115 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603818 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862885 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147459 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578597 s, 181 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144939 s, 116 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367263 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272084 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101578 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951657 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126331 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119592 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134932 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140854 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15738 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153606 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146219 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146334 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171117 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172372 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163104 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163912 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185049 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342069 s, 30.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275498 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240198 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170648 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294496 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202128 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213836 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286405 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377889 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300243 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163382 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155415 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389359 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387073 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246329 s, 42.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132419 s, 127 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385183 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592332 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113237 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908431 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133645 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139922 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124537 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161225 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133625 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158983 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147611 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158367 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16089 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105647 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171274 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165664 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027952 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118214 s, 88.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252069 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202459 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043582 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138369 s, 75.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102328 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271372 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688447 s, 152 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227268 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587392 s, 179 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178461 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971431 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250606 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189767 s, 55.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13787 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016248 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751007 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069557 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675173 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702584 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073931 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483633 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685894 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865066 s, 12.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901446 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898619 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770476 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193973 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123172 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144173 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130125 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474342 s, 221 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140528 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177329 s, 59.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223793 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142223 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197963 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030597 s, 343 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199113 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209687 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262847 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170187 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139614 s, 75.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193323 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141514 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396335 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470988 s, 223 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 05:06:44 (1766398004) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 05:06:48 (1766398008) oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 05:06:52 (1766398012) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 05:06:59 (1766398019) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.112374 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229437 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382094 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543418 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480599 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572384 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504582 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040778 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775463 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954621 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087682 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842818 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858961 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909494 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100119 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956474 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105027 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164759 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227476 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191986 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227952 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274288 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249876 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240367 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212206 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297866 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312072 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901977 s, 116 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320543 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346003 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398826 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887272 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169252 s, 62.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98137 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4385 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (54s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 05:07:53 (1766398073) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22345 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619543 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609179 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817635 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834553 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808288 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814559 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749859 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714159 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159505 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15948 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172279 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145681 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165686 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14757 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206711 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175217 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173536 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00678598 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139029 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204083 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112704 s, 93.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197944 s, 53.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258329 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224555 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261988 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119202 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497843 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446044 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00960125 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462927 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063713 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391618 s, 268 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8359 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9538 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23341 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105527 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11239 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107649 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129929 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142956 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144277 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123661 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139278 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120805 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12917 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171556 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173171 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177438 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155733 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184792 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1534 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226217 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270472 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280705 s, 37.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253646 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332808 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319042 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280069 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262878 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379228 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029762 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324291 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301651 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288036 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272687 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304575 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03629 s, 28.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8347 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.969 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (138s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 05:10:11 (1766398211) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22941 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877176 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765731 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889743 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735371 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794761 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752549 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780049 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932823 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157966 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157902 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177247 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140893 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14938 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172802 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182215 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17302 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189667 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142487 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206666 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214493 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210251 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220747 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133775 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173343 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250946 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222285 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022548 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235741 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214826 s, 48.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227267 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240738 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247614 s, 42.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8348 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9531 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22801 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123766 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966934 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113371 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115007 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144252 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146993 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118128 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112335 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123946 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151733 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131726 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134046 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176076 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176063 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185262 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183266 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219574 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156635 s, 66.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218442 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227356 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019003 s, 55.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244387 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247114 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288884 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022289 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271152 s, 38.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194574 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209344 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277026 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260329 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236339 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264418 s, 39.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8372 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.951 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (138s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 05:12:30 (1766398350) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.259417 s, 64.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295269 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313451 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616838 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061621 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883707 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783485 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780952 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618667 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687573 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104942 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915895 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105879 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10643 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827248 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106453 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120936 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179912 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225913 s, 46.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201395 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194353 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194018 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197518 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196215 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252955 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217073 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246152 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017631 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226732 s, 46.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264568 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023958 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240832 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269654 s, 38.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0813 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74909 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126634 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369843 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336756 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122424 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122127 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14628 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122581 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126941 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129141 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15052 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15482 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141824 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155281 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140189 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149569 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157457 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153474 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223857 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021483 s, 48.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200247 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198295 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209412 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02517 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218026 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267794 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255033 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884726 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357952 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266687 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280487 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00328834 s, 319 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360213 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523273 s, 200 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93034 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73288 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135599 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306797 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924687 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100841 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946415 s, 11.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937527 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109777 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120636 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10821 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11062 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124082 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118446 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119838 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113294 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989708 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103646 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14018 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182596 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187831 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220262 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968404 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199701 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00925142 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102932 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148414 s, 70.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319131 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178207 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175065 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154328 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958077 s, 109 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538247 s, 195 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130707 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110114 s, 95.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9726 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.8862 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12042 s, 139 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229842 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217394 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758402 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777133 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656085 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748029 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718602 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786701 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911533 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886658 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963248 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878897 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759715 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897657 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817042 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568893 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936286 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131502 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118041 s, 88.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110334 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183204 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168199 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229503 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146701 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166306 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024933 s, 42.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237128 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255835 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756249 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340192 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325783 s, 322 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349335 s, 300 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76856 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72961 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91442 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73497 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.847633 s, 19.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027992 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258138 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471001 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491766 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583745 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702022 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636733 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723033 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164631 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170411 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178768 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195148 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188686 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20959 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200861 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215943 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165885 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155132 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158501 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141087 s, 74.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183576 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163377 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568038 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168592 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208762 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457487 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180519 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157218 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904656 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379452 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389519 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426172 s, 246 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6739 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4397 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.24218 s, 69.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484798 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524957 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667304 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746917 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857561 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942945 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652312 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885197 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793261 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640491 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779416 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858939 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990306 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760356 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092151 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830907 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102781 s, 102 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143737 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139984 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018878 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214023 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157589 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018759 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204768 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107184 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380074 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019061 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441217 s, 238 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131184 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258503 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692582 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722581 s, 145 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94701 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7406 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131278 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271115 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515255 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433693 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694232 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693358 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531238 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751919 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587233 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816549 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778671 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710624 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692917 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100049 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999905 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956445 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693577 s, 15.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131686 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181934 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185464 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195395 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021321 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017114 s, 61.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169495 s, 61.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120193 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221865 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267257 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375503 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241343 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239872 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131025 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188591 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419035 s, 250 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7819 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7337 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76722 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73789 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (358s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 05:18:28 (1766398708) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253789 s, 66.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280222 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245744 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728794 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622441 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937342 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109807 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103346 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979394 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102479 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106867 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882509 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968376 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122752 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987397 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115902 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112104 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158698 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161713 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197118 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176919 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158753 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01678 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224536 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203899 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190265 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690331 s, 152 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956122 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625322 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00461295 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255745 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296235 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154492 s, 67.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93511 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73354 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (38s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 05:19:06 (1766398746) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116652 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040524 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320111 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774823 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109749 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111978 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135391 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136862 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14288 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146735 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154257 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1321 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157965 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124058 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140964 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14541 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140904 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131929 s, 79.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215746 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025048 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160277 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208266 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208843 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216382 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0077894 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202524 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202969 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146391 s, 71.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788094 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185041 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441336 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725318 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303232 s, 34.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94011 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73402 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 05:19:47 (1766398787) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.263882 s, 63.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246266 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530532 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589475 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500365 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472755 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526038 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046924 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609288 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444589 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806511 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919187 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120978 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117176 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12338 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114875 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114601 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126647 s, 82.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216528 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192675 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206693 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166332 s, 63.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263443 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237019 s, 44.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181053 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010174 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249098 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440963 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449733 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263322 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280571 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112376 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645478 s, 162 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98235 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97056 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 05:20:23 (1766398823) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.45311 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285202 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600223 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784495 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105257 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993953 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113092 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120768 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122196 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137377 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151583 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138162 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135342 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2003 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196136 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253145 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249402 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152625 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017286 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222734 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179969 s, 58.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196003 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125695 s, 83.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177914 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269222 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245172 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828321 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233147 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333276 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303613 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603515 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798777 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536741 s, 195 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83521 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73568 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93917 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73799 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 05:21:44 (1766398904) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.853917 s, 19.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262326 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514436 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341179 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515416 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539303 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758902 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687929 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593193 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159301 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18935 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188676 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208486 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210848 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189206 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195564 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102561 s, 102 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102761 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196532 s, 53.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168592 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159741 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115242 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160223 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182217 s, 57.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516143 s, 203 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100364 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273946 s, 38.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554393 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488829 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482715 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294681 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108469 s, 96.7 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6749 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9384 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 05:22:51 (1766398971) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254298 s, 66.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031315 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228645 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967109 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827492 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765778 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106596 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105136 s, 10.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111305 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970996 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963034 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105694 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937067 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100073 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102096 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108224 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928362 s, 11.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215445 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208928 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259863 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208835 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304043 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274617 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024895 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105904 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135233 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122806 s, 85.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185288 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245541 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363969 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255441 s, 41.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383866 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189956 s, 55.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03757 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73284 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.45756 s, 36.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559559 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752998 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112021 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121523 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152613 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180137 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145066 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157032 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151937 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154471 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133643 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167388 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196284 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183054 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317385 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319381 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154158 s, 68.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205482 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134048 s, 78.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154054 s, 68.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205829 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119771 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196407 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017672 s, 59.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202667 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777354 s, 135 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513977 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066824 s, 157 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231429 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00666302 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397506 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045851 s, 229 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78352 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73448 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77835 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73467 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (123s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 05:24:54 (1766399094) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 05:24:58 (1766399098) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg439-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.56436 s, 294 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg439-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0051 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 05:25:49 (1766399149) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138654 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362782 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784749 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851941 s, 12.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890341 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113124 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10695 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086831 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103237 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104131 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113556 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908232 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109436 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117001 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110304 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13337 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113891 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207403 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187441 s, 55.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125055 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163607 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152305 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151861 s, 69.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138938 s, 75.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332995 s, 315 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162149 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204243 s, 51.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f847fb8000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff93f847fb8000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029771 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918124 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00714615 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268375 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466866 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00849637 s, 123 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99992 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96905 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127712 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273903 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049802 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388215 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566176 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510524 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727609 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752913 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915226 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902552 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878162 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110142 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103403 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102549 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101011 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111311 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753621 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195896 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206049 s, 50.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242471 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019179 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217525 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166994 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208309 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023681 s, 44.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196844 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209287 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027864 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204661 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025792 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237503 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235295 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025426 s, 41.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93985 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97089 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 05:26:55 (1766399215) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 05:26:59 (1766399219) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (40s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 05:27:39 (1766399259) mds.MDS.mdt.nrs_policies=tbf oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg439-client: oleg439-server: ssh exited with exit code 36 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 05:27:51 (1766399271) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 310801 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 05:28:31 (1766399311) oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg439-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 05:28:34 (1766399314) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506442 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:28:35.000000000 -0500 Modify: 2025-12-22 05:28:35.000000000 -0500 Change: 2025-12-22 05:28:35.000000000 -0500 Birth: 2025-12-22 05:28:35.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 05:28:47 (1766399327) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 05:28:48 (1766399328) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 05:28:49 (1766399329) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 05:28:50 (1766399330) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 05:28:51 (1766399331) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 05:28:52 (1766399332) total: 2500 open/close in 5.28 seconds: 473.70 ops/second snapshot_time 1766399356.155952431 secs.nsecs start_time 1766399333.929281037 secs.nsecs elapsed_time 22.226671394 secs.nsecs open 2505 samples [usecs] 114 1686 739422 237375698 close 2504 samples [usecs] 67 1021 239368 26228850 mknod 2501 samples [usecs] 172 1198 641395 179155329 unlink 2500 samples [usecs] 277 17376 1011795 940154427 mkdir 2 samples [usecs] 685 1113 1798 1707994 rmdir 2 samples [usecs] 1870 2504 4374 9766916 rename 2500 samples [usecs] 257 15677 3050075 5480060295 getattr 3760 samples [usecs] 8 1551 80648 7086318 parallel_rename_file 2500 samples [usecs] 257 15677 3050075 5480060295 crossdir_rename 2500 samples [usecs] 257 15677 3050075 5480060295 PASS 81d (25s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 05:29:17 (1766399357) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7565 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7565 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 05:29:20 (1766399360) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 05:29:21 (1766399361) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 05:29:28 (1766399368) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1766399369 Finishing test test1 at 1766399369 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 05:29:31 (1766399371) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 05:29:32 (1766399372) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:29:33 (1766399373) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:29:34 (1766399374) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5763 0x1683 0x280000400 0 5789 0x169d 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5764 0x1684 0x280000400 0 5790 0x169e 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 05:29:43 (1766399383) osc.lustre-OST0000-osc-ffff93f847fb8000.idle_timeout=0 osc.lustre-OST0000-osc-ffff93f8483b5000.idle_timeout=0 osc.lustre-OST0001-osc-ffff93f847fb8000.idle_timeout=0 osc.lustre-OST0001-osc-ffff93f8483b5000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0370194 s, 221 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09104 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 328928 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0114252 s, 359 kB/s osc.lustre-OST0000-osc-ffff93f847fb8000.idle_timeout=debug osc.lustre-OST0000-osc-ffff93f8483b5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f847fb8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f8483b5000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 05:29:54 (1766399394) llite.lustre-ffff93f847fb8000.fast_read=0 llite.lustre-ffff93f8483b5000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.022262 s, 94.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff93f847fb8000.fast_read=1 llite.lustre-ffff93f8483b5000.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:30:08 (1766399408) llite.lustre-ffff93f847fb8000.fast_read=0 llite.lustre-ffff93f8483b5000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00167823 s, 7.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff93f847fb8000.fast_read=1 llite.lustre-ffff93f8483b5000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:30:16 (1766399416) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:30:17 (1766399417) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00141791 s, 2.9 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:30:20 (1766399420) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0457134 s, 45.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:30:22 (1766399422) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0628632 s, 33.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 05:30:24 (1766399424) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00205238 s, 5.8 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 05:30:26 (1766399426) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000470182 s, 8.7 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 05:30:28 (1766399428) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000440662 s, 9.3 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 05:30:30 (1766399430) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000474236 s, 8.6 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 05:30:34 (1766399434) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1766399435 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1766399437 Finishing test test23 at 1766399438 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:30:41 (1766399441) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 05:30:42 (1766399442) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 05:31:05 (1766399465) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:31:06 (1766399466) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00395468 s, 259 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508977 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:31:07.000000000 -0500 Modify: 2025-12-22 05:31:07.000000000 -0500 Change: 2025-12-22 05:31:07.000000000 -0500 Birth: 2025-12-22 05:31:07.000000000 -0500 /mnt/lustre/f106b.sanityn 144115205272508977 -rw-r--r-- 1 0 0 1766399467 1766399467 1766399467 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508977 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:31:07.000000000 -0500 Modify: 2025-12-22 05:31:07.000000000 -0500 Change: 2025-12-22 05:31:07.000000000 -0500 Birth: 2025-12-22 05:31:07.000000000 -0500 1766399467 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 05:31:09 (1766399469) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 05:31:11 (1766399471) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.104386 s, 100 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5798 0x16a6 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:31:16 (1766399476) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0943163 s, 111 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5771 0x168b 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7565 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 05:31:24 (1766399484) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00180624 s, 5.5 MB/s llite.lustre-ffff93f847fb8000.hybrid_io=0 llite.lustre-ffff93f8483b5000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.34124 s, 1.9 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:31:30 (1766399490) 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 PASS 109 (508s) == sanityn test 110: do not grant another lock on resend ========================================================== 05:39:58 (1766399998) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 05:39:58.000000000 -0500 Modify: 2025-12-22 05:39:58.000000000 -0500 Change: 2025-12-22 05:39:58.000000000 -0500 Birth: 2025-12-22 05:39:59.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg439-client.virtnet oleg439-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 05:41:03 (1766400063) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 05:41:04 (1766400064) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 05:41:05 (1766400065) lustre-OST0000-osc-ffff93f843ef7000 active. lustre-OST0001-osc-ffff93f843ef7000 active. lustre-OST0000-osc-ffff93f843ef7000 active. lustre-OST0001-osc-ffff93f843ef7000 active. MGC192.168.204.139@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 05:41:08 (1766400068) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 05:41:09 (1766400069) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 05:41:10 (1766400070) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 05:41:11 (1766400071) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (3s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 05:41:14 (1766400074) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 05:41:15 (1766400075) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (1s) == sanityn test 117d: Per-User Trash can Type testing ==== 05:41:16 (1766400076) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 05:41:17 (1766400077) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (1s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 05:41:18 (1766400078) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 05:41:19 (1766400079) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 05:41:20 (1766400080) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (5s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 05:41:25 (1766400085) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (3s) == sanityn test 117j: Move back file from Trash Can ====== 05:41:28 (1766400088) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 05:41:29 (1766400089) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 05:41:30 (1766400090) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (1s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 05:41:31 (1766400091) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (1s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 05:41:32 (1766400092) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (1s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 05:41:33 (1766400093) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (1s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 05:41:34 (1766400094) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 05:41:35 (1766400095) SKIP: sanityn test_117pb needs >= 2 MDTs SKIP 117pb (1s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 05:41:36 (1766400096) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (1s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 05:41:37 (1766400097) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (1s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 05:41:38 (1766400098) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (1s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 05:41:39 (1766400099) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (2s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 05:41:41 (1766400101) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (1s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 05:41:42 (1766400102) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 05:41:43 (1766400103) SKIP: sanityn test_117v ldiskfs only test SKIP 117v (2s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 05:41:45 (1766400105) SKIP: sanityn test_117w ldiskfs only test SKIP 117w (1s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 05:41:46 (1766400106) SKIP: sanityn test_117x ldiskfs only test SKIP 117x (1s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 05:41:47 (1766400107) SKIP: sanityn test_117ya ldiskfs only test SKIP 117ya (2s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 05:41:49 (1766400109) SKIP: sanityn test_117yb ldiskfs only test SKIP 117yb (1s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 05:41:50 (1766400110) SKIP: sanityn test_117yc ldiskfs only test SKIP 117yc (1s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 05:41:51 (1766400111) SKIP: sanityn test_117za ldiskfs only test SKIP 117za (1s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 05:41:52 (1766400112) SKIP: sanityn test_117zb ldiskfs only test SKIP 117zb (1s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 05:41:53 (1766400113) SKIP: sanityn test_117zc ldiskfs only test SKIP 117zc (1s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 05:41:54 (1766400114) On MGS 192.168.204.139, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.204.139, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] SKIP: sanityn test_117zd ldiskfs only test On MGS 192.168.204.139, active = nodemap.active=0 waiting 10 secs for sync SKIP 117zd (37s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 05:42:31 (1766400151) SKIP: sanityn test_118Aa ldiskfs only test SKIP 118Aa (2s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 05:42:33 (1766400153) SKIP: sanityn test_118Ab ldiskfs only test SKIP 118Ab (1s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 05:42:34 (1766400154) SKIP: sanityn test_118Ba ldiskfs only test SKIP 118Ba (1s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 05:42:35 (1766400155) SKIP: sanityn test_118Bb ldiskfs only test SKIP 118Bb (2s) == sanityn test 118C: TCU: add support for rename() ====== 05:42:37 (1766400157) SKIP: sanityn test_118C ldiskfs only test SKIP 118C (1s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 05:42:38 (1766400158) SKIP: sanityn test_118d ldiskfs only test SKIP 118d (2s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 05:42:40 (1766400160) SKIP: sanityn test_118e ldiskfs only test SKIP 118e (1s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 05:42:41 (1766400161) SKIP: sanityn test_118f ldiskfs only test SKIP 118f (1s) == sanityn test 118g: TCU: Tolerate ENOSPC error bypass Trash Can ========================================================== 05:42:42 (1766400162) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 562023 14471 547552 3% /mnt/lustre[MDT:0] lustre-OST0000_UUID 119366 2438 116928 3% /mnt/lustre[OST:0] lustre-OST0001_UUID 118423 2423 116000 3% /mnt/lustre[OST:1] filesystem_summary: 247399 14471 232928 6% /mnt/lustre - create 8898 (time 1766400175.88 total 10.02 last 887.97) - create 17159 (time 1766400185.88 total 20.02 last 826.00) - create 20000 (time 1766400189.89 total 24.03 last 708.80) - create 28005 (time 1766400199.89 total 34.03 last 800.48) - create 30000 (time 1766400202.45 total 36.60 last 777.50) - create 38856 (time 1766400212.45 total 46.60 last 885.54) - create 47907 (time 1766400222.45 total 56.60 last 905.07) - create 50000 (time 1766400224.62 total 58.76 last 967.89) - create 58440 (time 1766400234.62 total 68.76 last 843.94) - create 66870 (time 1766400244.62 total 78.76 last 842.94) - create 70000 (time 1766400248.60 total 82.74 last 786.68) - create 78242 (time 1766400258.60 total 92.74 last 824.13) - create 86342 (time 1766400268.60 total 102.74 last 809.99) - create 90000 (time 1766400272.49 total 106.63 last 940.44) - create 98390 (time 1766400282.49 total 116.63 last 838.96) - create 100000 (time 1766400285.06 total 119.20 last 625.55) total: 100707 create in 120.00 seconds: 839.22 ops/second - unlinked 0 (time 1766400287 ; total 0 ; last 0) - unlinked 10000 (time 1766400311 ; total 24 ; last 24) - unlinked 20000 (time 1766400334 ; total 47 ; last 23) - unlinked 30000 (time 1766400359 ; total 72 ; last 25) - unlinked 40000 (time 1766400385 ; total 98 ; last 26) - unlinked 50000 (time 1766400411 ; total 124 ; last 26) - unlinked 60000 (time 1766400437 ; total 150 ; last 26) - unlinked 70000 (time 1766400460 ; total 173 ; last 23) - unlinked 80000 (time 1766400483 ; total 196 ; last 23) - unlinked 90000 (time 1766400506 ; total 219 ; last 23) - unlinked 100000 (time 1766400533 ; total 246 ; last 27) unlink(/mnt/lustre/d118g.sanityn/tf100707) error: No such file or directory total: 100707 unlinks in 249 seconds: 404.445770 unlinks/second - create 6590 (time 1766400910.22 total 10.00 last 658.93) - create 10000 (time 1766400915.97 total 15.75 last 593.50) - create 15819 (time 1766400925.97 total 25.75 last 581.82) - create 20000 (time 1766400933.39 total 33.16 last 563.86) - create 25396 (time 1766400943.39 total 43.16 last 539.59) - create 30000 (time 1766400952.64 total 52.42 last 497.35) - create 35478 (time 1766400962.64 total 62.42 last 547.78) - create 40000 (time 1766400971.16 total 70.93 last 531.14) - create 45178 (time 1766400981.16 total 80.93 last 517.77) - create 50000 (time 1766400989.78 total 89.56 last 559.16) - create 55755 (time 1766400999.78 total 99.56 last 575.42) - create 60000 (time 1766401007.63 total 107.40 last 541.17) - create 67052 (time 1766401017.63 total 117.41 last 705.13) total: 69100 create in 120.00 seconds: 575.82 ops/second - unlinked 0 (time 1766401024 ; total 0 ; last 0) - unlinked 10000 (time 1766401060 ; total 36 ; last 36) - unlinked 20000 (time 1766401092 ; total 68 ; last 32) - unlinked 30000 (time 1766401114 ; total 90 ; last 22) - unlinked 40000 (time 1766401136 ; total 112 ; last 22) - unlinked 50000 (time 1766401163 ; total 139 ; last 27) - unlinked 60000 (time 1766401192 ; total 168 ; last 29) unlink(/mnt/lustre/d118g.sanityn/tf69100) error: No such file or directory total: 69100 unlinks in 199 seconds: 347.236176 unlinks/second mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid lfs trash trash: failed to open by FID:[0x200000002:0x4:0x0]: Stale file handle (116) test_118g returned 116 FAIL 118g (1063s) == sanityn test 118h: TCU: run mdtest with TCU enabled === 06:00:27 (1766401227) SKIP: sanityn test_118h mdtest not found SKIP 118h (1s) == sanityn test 118i: TCU: trusted.unrm xattr should not be cached on client-side ========================================================== 06:00:28 (1766401228) SKIP: sanityn test_118i ldiskfs only test SKIP 118i (2s) == sanityn test 200: service remains healthy while able to process request ========================================================== 06:00:30 (1766401230) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=19 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0941 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0984 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1238 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1158 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1138 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1119 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1195 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1027 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.124 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1478 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.156 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1498 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1911 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1886 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.176 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1744 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1854 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1573 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2215 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2545 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2416 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2302 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2323 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2813 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2365 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2701 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2297 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3249 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3514 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3166 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3455 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3261 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3621 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3643 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3588 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3378 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.422 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.43 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93f843ef7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f843ef7000.ost_server_uuid in FULL state after 0 sec PASS 200 (109s) cleanup: ====================================================== == sanityn test complete, duration 7254 sec ============== 06:02:19 (1766401339) === sanityn: start cleanup 06:02:20 (1766401340) === Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 06:03:27 (1766401407) ===