-----============= acceptance-small: sanityn ============----- Wed Sep 24 13:07:35 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg330-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 13:07:44 (1758733664) === Starting client oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Started clients oleg330-client.virtnet: 192.168.203.130@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg330-client.virtnet: executing check_config_client /mnt/lustre oleg330-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg330-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d17881f1000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d179dd36800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d17881f1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d179dd36800.idle_timeout=debug disable quota as required oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 13:08:04 (1758733684) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015423 s, 68.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7550] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 13:08:06 (1758733686) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 13:08:11 (1758733691) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 13:08:16 (1758733696) -rw-r--r-- 1 root root 0 Sep 24 13:08 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 13:08:20 (1758733700) -rw-r--r-- 1 root root 0 Sep 24 13:08 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 13:08:24 (1758733704) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 13:08:27 (1758733707) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 13:08:31 (1758733711) total 1 -rwxrwxrwx 1 root root 0 Sep 24 13:08 f2a -rwxrwxrwx 1 root root 0 Sep 24 13:08 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 24 13:08 f2a -rwxrwxrwx 1 root root 0 Sep 24 13:08 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00578816 s, 88.5 kB/s PASS 2e (5s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 13:08:36 (1758733716) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 13:08:38 (1758733718) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190121 s, 11.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 13:08:42 (1758733722) PASS 3 (4s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 13:08:46 (1758733726) PASS 4 (5s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 13:08:51 (1758733731) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (4s) == sanityn test 6: remove of open file on other node ============================================================================ 13:08:55 (1758733735) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (4s) == sanityn test 7: remove of open directory on other node ======================================================================= 13:08:59 (1758733739) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 13:09:03 (1758733743) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 13:09:08 (1758733748) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 13:09:14 (1758733754) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526131 s, 19.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0573753 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 13:09:19 (1758733759) 1+0 records in 1+0 records out 1 byte copied, 0.00782581 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112341 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204437 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0137423 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0139314 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118442 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129364 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104881 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0168781 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0106844 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0162224 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0215676 s, 0.0 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 13:09:25 (1758733765) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0113169 s, 271 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0271515 s, 151 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000452002 s, 6.8 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 13:09:30 (1758733770) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (5s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 13:09:35 (1758733775) warning: '-runtime' deprecated, use '-t runtime' instead - link 459 (time 1758733787.36 total 10.03 last 45.77) - link 932 (time 1758733797.36 total 20.03 last 47.30) - link 1403 (time 1758733807.37 total 30.04 last 47.06) - link 1927 (time 1758733817.39 total 40.05 last 52.31) - link 2471 (time 1758733827.39 total 50.06 last 54.39) - link 3129 (time 1758733837.42 total 60.08 last 65.61) - link 3693 (time 1758733847.44 total 70.10 last 56.29) - link 4175 (time 1758733857.44 total 80.11 last 48.17) - link 4737 (time 1758733867.45 total 90.12 last 56.16) total: 5307 link in 100.01 seconds: 53.07 ops/second using seed 278964438 running for 100 seconds - stat 10000 (time 1758733788 ; total 10 ; last 10) - stat 19676 (time 1758733799 ; total 21 ; last 11) - stat 20000 (time 1758733799 ; total 21 ; last 0) - stat 30000 (time 1758733809 ; total 31 ; last 10) - stat 40000 (time 1758733819 ; total 41 ; last 10) - stat 50000 (time 1758733828 ; total 50 ; last 9) - stat 55001 (time 1758733839 ; total 61 ; last 11) - stat 60000 (time 1758733849 ; total 71 ; last 10) - stat 64586 (time 1758733860 ; total 82 ; last 11) - stat 69546 (time 1758733871 ; total 93 ; last 11) - stat 70000 (time 1758733871 ; total 93 ; last 0) total: 73502 stats in 100 seconds: 735.020020 stats/second - unlinked 0 (time 1758733828 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile5307) error: No such file or directory total: 5306 unlinks in 93 seconds: 57.053764 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21334) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21336) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 13:12:04 (1758733924) f13 f13 PASS 13 (6s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 13:12:10 (1758733930) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 22521 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (4s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 13:12:14 (1758733934) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 23100 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (4s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 13:12:18 (1758733938) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23680 Terminated $DIR1/$tdir/sleep 60 PASS 14b (5s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 13:12:23 (1758733943) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24262 Terminated $DIR1/$tdir/sleep 60 PASS 14c (4s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 13:12:27 (1758733947) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24844 Terminated $DIR1/$tdir/sleep 60 PASS 14d (5s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 13:12:32 (1758733952) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7526400 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7526400KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (39s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 13:13:11 (1758733991) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.276134 s, 30.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.253461 s, 33.1 MB/s Chance of close/open is 1 in 50 Seed set to 3993 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000006[0] 1758733994.008789 mapwrite 0x33d95e thru 0x345132 (0x77d5 bytes) truncating to largest ever: 0x7580a0 000007[0] 1758733994.122434 trunc from 0x345133 to 0x75809f (0x412f6d bytes) 000008[0] 1758733994.135553 write 0x3b9927 thru 0x3c2c78 (0x9352 bytes) 000009[0] 1758733994.147574 write 0x1e7f34 thru 0x1ed3d2 (0x549f bytes) 000010[1] 1758733994.175443 read 0x31dfbb thru 0x328a9a (0xaae0 bytes) truncating to largest ever: 0x7d7200 000011[1] 1758733994.228554 trunc from 0x7580a0 to 0x7d71ff (0x7f160 bytes) 000012[1] 1758733994.259277 write 0x9a1653 thru 0x9a333f (0x1ced bytes) 000015[0] 1758733994.296628 mapread 0x90ede2 thru 0x91a71c (0xb93b bytes) 000021[1] 1758733994.398853 write 0x02f53d thru 0x03ae26 (0xb8ea bytes) 000023[1] 1758733994.449341 trunc from 0x9a3340 to 0x635d65 (0x36d5da bytes) 000024[0] 1758733994.474134 write 0x0a715a thru 0x0b3226 (0xc0cd bytes) 000025[0] 1758733994.524517 write 0x05c315 thru 0x05f786 (0x3472 bytes) 000026[1] 1758733994.548538 mapwrite 0x9255ee thru 0x930eb6 (0xb8c9 bytes) 000027[0] 1758733994.659460 write 0x679c37 thru 0x67cbd7 (0x2fa1 bytes) 000028[1] 1758733994.689030 read 0x6ab368 thru 0x6af2ae (0x3f47 bytes) 000029[1] 1758733994.718705 write 0x828a71 thru 0x82a510 (0x1aa0 bytes) 000031[1] 1758733994.763987 write 0x4459ba thru 0x44b993 (0x5fda bytes) 000036[0] 1758733994.847207 trunc from 0x930eb7 to 0x10694d (0x82a569 bytes) 000040[0] 1758733994.895429 write 0x65a69c thru 0x65ee7a (0x47df bytes) 000041[0] 1758733994.919935 mapread 0x355441 thru 0x35c8ee (0x74ae bytes) 000043[0] 1758733995.006984 trunc from 0x65ee7b to 0x1b8dec (0x4a608e bytes) 000044[1] 1758733995.030972 mapwrite 0x34b995 thru 0x34fbd6 (0x4242 bytes) 000045[0] 1758733995.153297 write 0x7692fd thru 0x76db44 (0x4848 bytes) 000049[1] 1758733995.232031 trunc from 0x76db45 to 0x6f6761 (0x773e3 bytes) 000050[1] 1758733995.256157 write 0x0a600f thru 0x0ac40f (0x6401 bytes) 000055[0] 1758733995.319819 write 0x87c6b9 thru 0x88a13c (0xda84 bytes) 000057[0] 1758733995.363906 mapwrite 0x569675 thru 0x56b2c4 (0x1c50 bytes) 000058[1] 1758733995.449704 read 0x4b4e98 thru 0x4c2622 (0xd78b bytes) 000059[0] 1758733995.490661 mapread 0x580dcd thru 0x588aff (0x7d33 bytes) 000062[1] 1758733995.520152 write 0x08b54b thru 0x0938c1 (0x8377 bytes) 000064[1] 1758733995.560424 trunc from 0x88a13d to 0x43f590 (0x44abac bytes) 000067[0] 1758733995.657643 mapwrite 0x40a826 thru 0x411c49 (0x7424 bytes) 000070[1] 1758733995.799185 write 0x2fe0c9 thru 0x30b37d (0xd2b5 bytes) 000073[0] 1758733995.853381 read 0x0c2217 thru 0x0c560a (0x33f4 bytes) 000074[0] 1758733995.892334 write 0x3bb976 thru 0x3c7904 (0xbf8f bytes) 000076[0] 1758733995.977439 trunc from 0x43f591 to 0x20d6cf (0x231ec1 bytes) 000078[1] 1758733996.010803 mapwrite 0x8e5b69 thru 0x8e9d10 (0x41a8 bytes) 000079[0] 1758733996.117697 mapwrite 0x049ebc thru 0x04ced6 (0x301b bytes) 000081[1] 1758733996.233205 mapwrite 0x63db2f thru 0x646c71 (0x9143 bytes) 000082[0] 1758733996.306194 write 0x416bb3 thru 0x41aca1 (0x40ef bytes) 000083[1] 1758733996.328696 write 0x24d2af thru 0x25400e (0x6d60 bytes) 000084[0] 1758733996.357540 mapwrite 0x081cde thru 0x08b6dd (0x9a00 bytes) 000085[0] 1758733996.445048 mapwrite 0x6604fe thru 0x66779b (0x729e bytes) 000086[0] 1758733996.521186 mapread 0x051151 thru 0x055a11 (0x48c1 bytes) 000087[1] 1758733996.582491 trunc from 0x8e9d11 to 0x1b420d (0x735b03 bytes) 000088[1] 1758733996.630297 trunc from 0x1b420e to 0x1c9dea (0x15bdd bytes) truncating to largest ever: 0x7e4103 000091[0] 1758733996.661159 trunc from 0x1c9deb to 0x7e4102 (0x61a318 bytes) 000094[0] 1758733996.766576 mapread 0x2c4996 thru 0x2d194a (0xcfb5 bytes) 000097[0] 1758733996.835588 mapwrite 0x7ffbaa thru 0x80c30a (0xc761 bytes) 000098[1] 1758733996.977476 write 0x8c2a66 thru 0x8c4878 (0x1e13 bytes) 000101[0] 1758733997.039269 write 0x19b212 thru 0x1a85b6 (0xd3a5 bytes) 000103[1] 1758733997.139188 mapwrite 0x3cb876 thru 0x3d5e0b (0xa596 bytes) 000104[1] 1758733997.247252 read 0x192b8c thru 0x19c5cf (0x9a44 bytes) 000105[1] 1758733997.287080 mapwrite 0x1a4803 thru 0x1a9591 (0x4d8f bytes) 000106[0] 1758733997.357115 write 0x713588 thru 0x716da7 (0x3820 bytes) 000107[0] 1758733997.384198 read 0x82782d thru 0x82f6fc (0x7ed0 bytes) 000108[0] 1758733997.408458 write 0x6416be thru 0x64d55b (0xbe9e bytes) 000109[0] 1758733997.434598 read 0x86fed9 thru 0x87ae11 (0xaf39 bytes) 000111[1] 1758733997.451716 mapwrite 0x246446 thru 0x24b066 (0x4c21 bytes) 000112[1] 1758733997.548204 read 0x17e050 thru 0x18b466 (0xd417 bytes) 000116[1] 1758733997.635705 read 0x37a2a3 thru 0x387d29 (0xda87 bytes) 000119[1] 1758733997.703398 mapread 0x07daf6 thru 0x084756 (0x6c61 bytes) 000120[1] 1758733997.750149 mapwrite 0x3ce466 thru 0x3d2ef5 (0x4a90 bytes) 000122[1] 1758733997.832457 trunc from 0x8c4879 to 0x1b38b5 (0x710fc3 bytes) 000125[1] 1758733997.871426 write 0x7de965 thru 0x7e4e6c (0x6508 bytes) 000126[0] 1758733997.904815 read 0x5ef5ae thru 0x5f452e (0x4f81 bytes) 000128[1] 1758733997.967066 mapread 0x4a333d thru 0x4b04c8 (0xd18c bytes) 000130[1] 1758733998.044932 write 0x4a735b thru 0x4ad88b (0x6531 bytes) 000133[1] 1758733998.164436 mapwrite 0x266e7a thru 0x26aa09 (0x3b90 bytes) 000134[0] 1758733998.256432 mapwrite 0x1deba3 thru 0x1ee9af (0xfe0d bytes) 000135[1] 1758733998.355426 write 0x6177f8 thru 0x6240d0 (0xc8d9 bytes) 000136[1] 1758733998.382351 mapread 0x7aaf53 thru 0x7b04dc (0x558a bytes) 000142[1] 1758733998.527896 trunc from 0x7e4e6d to 0x5c5ec7 (0x21efa5 bytes) 000147[1] 1758733998.609679 mapwrite 0x8b3433 thru 0x8bc20e (0x8ddc bytes) 000148[0] 1758733998.719119 mapwrite 0x372086 thru 0x3801ba (0xe135 bytes) 000149[1] 1758733998.848365 write 0x7fbd44 thru 0x8096ff (0xd9bc bytes) 000150[1] 1758733998.902658 read 0x43376a thru 0x4394ea (0x5d81 bytes) 000151[1] 1758733998.938627 write 0x65c188 thru 0x65f2e1 (0x315a bytes) 000153[1] 1758733998.978765 read 0x063e11 thru 0x06954d (0x573d bytes) 000154[0] 1758733998.994402 read 0x6071f8 thru 0x61003b (0x8e44 bytes) 000158[1] 1758733999.068740 write 0x6f18e0 thru 0x700a58 (0xf179 bytes) 000164[0] 1758733999.183358 mapwrite 0x290717 thru 0x29b5b4 (0xae9e bytes) 000165[0] 1758733999.344778 mapread 0x1bb457 thru 0x1c4714 (0x92be bytes) 000168[0] 1758733999.409355 mapwrite 0x7c70b4 thru 0x7ce40d (0x735a bytes) 000169[1] 1758733999.547412 trunc from 0x8bc20f to 0x4d459b (0x3e7c73 bytes) 000171[0] 1758733999.633837 mapwrite 0x091439 thru 0x09ec74 (0xd83c bytes) 000172[0] 1758733999.764537 mapread 0x32f8a4 thru 0x336798 (0x6ef5 bytes) truncating to largest ever: 0x88b0e9 000174[1] 1758733999.834389 trunc from 0x4d459c to 0x88b0e8 (0x3b6b4d bytes) truncating to largest ever: 0x988ae6 000179[0] 1758733999.949324 trunc from 0x88b0e9 to 0x988ae5 (0xfd9fd bytes) 000181[1] 1758734000.001706 mapread 0x7c6121 thru 0x7ca8f8 (0x47d8 bytes) 000183[1] 1758734000.052097 read 0x384568 thru 0x3908eb (0xc384 bytes) 000184[1] 1758734000.074942 mapwrite 0x616548 thru 0x625013 (0xeacc bytes) 000185[1] 1758734000.196710 read 0x48b736 thru 0x490508 (0x4dd3 bytes) 000186[1] 1758734000.227718 write 0x2ab5b3 thru 0x2b06e1 (0x512f bytes) 000189[1] 1758734000.315737 mapwrite 0x60c0a8 thru 0x61bc7f (0xfbd8 bytes) 000192[0] 1758734000.400388 mapread 0x4037dd thru 0x409262 (0x5a86 bytes) 000193[1] 1758734000.459509 trunc from 0x988ae6 to 0x0bd6e2 (0x8cb403 bytes) 000194[0] 1758734000.513297 read 0x08c0f0 thru 0x08eefb (0x2e0c bytes) 000198[1] 1758734000.604728 mapread 0x05a72f thru 0x0668dc (0xc1ae bytes) 000199[0] 1758734000.625723 trunc from 0x0bd6e3 to 0x201578 (0x143e96 bytes) 000201[0] 1758734000.694154 mapread 0x09141d thru 0x094487 (0x306b bytes) 000202[0] 1758734000.771108 read 0x06ebb3 thru 0x07bd3b (0xd189 bytes) 000203[1] 1758734000.807921 write 0x5e17ec thru 0x5e22ef (0xb04 bytes) 000204[0] 1758734000.845059 trunc from 0x5e22f0 to 0x0b7dec (0x52a503 bytes) 000205[0] 1758734000.892425 read 0x059ff4 thru 0x05a728 (0x735 bytes) 000207[1] 1758734000.898630 mapread 0x01759e thru 0x020c67 (0x96ca bytes) 000209[0] 1758734000.972964 read 0x05db84 thru 0x06d8c7 (0xfd44 bytes) 000210[1] 1758734000.997143 trunc from 0x0b7ded to 0x1e7c83 (0x12fe97 bytes) 000212[0] 1758734001.056278 mapread 0x0c44f3 thru 0x0c7e83 (0x3991 bytes) 000215[0] 1758734001.127733 mapread 0x1b29e4 thru 0x1bbccd (0x92ea bytes) 000217[0] 1758734001.160970 trunc from 0x1e7c84 to 0x932a97 (0x74ae14 bytes) 000219[0] 1758734001.246155 trunc from 0x932a98 to 0x4a9713 (0x489384 bytes) 000220[1] 1758734001.281657 mapwrite 0x2436e4 thru 0x247c95 (0x45b2 bytes) 000222[0] 1758734001.424379 mapread 0x434128 thru 0x43ee4d (0xad26 bytes) 000224[0] 1758734001.504484 mapread 0x2d787b thru 0x2db6c8 (0x3e4e bytes) 000226[1] 1758734001.592436 read 0x455a5c thru 0x45fdf1 (0xa396 bytes) 000227[0] 1758734001.613674 mapwrite 0x91c36c thru 0x9296e7 (0xd37c bytes) 000230[0] 1758734001.778483 mapwrite 0x6f4c1c thru 0x6f5fc8 (0x13ad bytes) 000233[1] 1758734001.861058 read 0x630275 thru 0x63771f (0x74ab bytes) 000234[1] 1758734001.894170 mapwrite 0x8fb33b thru 0x8fc8d8 (0x159e bytes) 000235[0] 1758734001.973837 read 0x04dc82 thru 0x05d3b5 (0xf734 bytes) 000236[1] 1758734001.998099 trunc from 0x9296e8 to 0x848b9f (0xe0b48 bytes) 000237[0] 1758734002.054450 trunc from 0x848ba0 to 0x10ea87 (0x73a118 bytes) 000239[0] 1758734002.098698 write 0x54e0ad thru 0x55c632 (0xe586 bytes) 000241[0] 1758734002.151061 mapread 0x19ff8d thru 0x1a87e1 (0x8855 bytes) 000246[1] 1758734002.271234 read 0x209973 thru 0x21885a (0xeee8 bytes) 000248[0] 1758734002.340816 read 0x2ea0e0 thru 0x2f0a78 (0x6999 bytes) 000249[0] 1758734002.364978 mapread 0x55966f thru 0x55c632 (0x2fc4 bytes) 000250[0] 1758734002.375692 mapwrite 0x887d24 thru 0x88d423 (0x5700 bytes) 000251[0] 1758734002.463718 mapwrite 0x93a9c0 thru 0x93e2a3 (0x38e4 bytes) 000252[0] 1758734002.530094 trunc from 0x93e2a4 to 0x3bb092 (0x583211 bytes) 000253[0] 1758734002.567540 trunc from 0x3bb093 to 0x62daf4 (0x272a62 bytes) 000254[1] 1758734002.586057 mapwrite 0x34906d thru 0x353aa2 (0xaa36 bytes) 000255[1] 1758734002.688972 mapread 0x58dfff thru 0x59dd87 (0xfd89 bytes) 000258[0] 1758734002.756413 write 0x28ab80 thru 0x28fb5d (0x4fde bytes) 000262[1] 1758734002.796676 read 0x41e5c9 thru 0x42645b (0x7e93 bytes) 000265[0] 1758734002.883117 write 0x011e82 thru 0x0205a1 (0xe720 bytes) 000266[0] 1758734002.900845 trunc from 0x62daf5 to 0x18d69d (0x4a0457 bytes) 000267[0] 1758734002.964224 read 0x0e6210 thru 0x0f5634 (0xf425 bytes) 000270[0] 1758734003.032329 write 0x8e0135 thru 0x8eb268 (0xb134 bytes) 000271[0] 1758734003.053217 write 0x3f807d thru 0x402c48 (0xabcc bytes) 000273[0] 1758734003.127574 mapwrite 0x63f86a thru 0x642c80 (0x3417 bytes) 000275[1] 1758734003.208663 trunc from 0x8eb269 to 0x940376 (0x5510e bytes) 000276[1] 1758734003.302715 trunc from 0x940377 to 0x951ed4 (0x11b5e bytes) 000277[0] 1758734003.320438 mapread 0x280b57 thru 0x28f8be (0xed68 bytes) 000278[1] 1758734003.460198 read 0x6c060b thru 0x6cfe22 (0xf818 bytes) 000282[0] 1758734003.525113 write 0x9aa399 thru 0x9adfd3 (0x3c3b bytes) 000288[1] 1758734003.636747 mapwrite 0x5ad300 thru 0x5afc38 (0x2939 bytes) 000290[1] 1758734003.740803 mapwrite 0x2f5287 thru 0x2ff867 (0xa5e1 bytes) 000293[1] 1758734003.861320 mapwrite 0x7a7300 thru 0x7a8164 (0xe65 bytes) 000295[1] 1758734003.925776 write 0x282384 thru 0x28d04a (0xacc7 bytes) 000296[1] 1758734003.935190 write 0x52b471 thru 0x52c30e (0xe9e bytes) 000297[0] 1758734003.948825 mapwrite 0x80547f thru 0x807e39 (0x29bb bytes) 000302[1] 1758734004.120621 mapwrite 0x00223e thru 0x00d9c2 (0xb785 bytes) 000303[0] 1758734004.207508 mapwrite 0x3034ca thru 0x30f450 (0xbf87 bytes) 000304[1] 1758734004.348840 write 0x97d31c thru 0x9854c0 (0x81a5 bytes) 000309[1] 1758734004.418436 write 0x5a0536 thru 0x5a252d (0x1ff8 bytes) 000310[1] 1758734004.450447 mapwrite 0x3cd2b9 thru 0x3cf993 (0x26db bytes) 000312[1] 1758734004.543032 trunc from 0x9adfd4 to 0x025a65 (0x98856e bytes) 000313[1] 1758734004.589467 read 0x012448 thru 0x012ebe (0xa77 bytes) 000314[0] 1758734004.603699 mapwrite 0x17481e thru 0x17ee55 (0xa638 bytes) 000315[1] 1758734004.729035 trunc from 0x17ee56 to 0x6a23e3 (0x52358e bytes) 000317[1] 1758734004.808704 read 0x504fc8 thru 0x509200 (0x4239 bytes) 000318[0] 1758734004.820022 mapread 0x26b539 thru 0x271b7a (0x6642 bytes) 000320[1] 1758734004.877558 mapwrite 0x930d6e thru 0x93362c (0x28bf bytes) 000321[1] 1758734004.962826 read 0x19e625 thru 0x1add16 (0xf6f2 bytes) 000322[0] 1758734005.001845 read 0x342794 thru 0x34ffc5 (0xd832 bytes) 000324[1] 1758734005.035165 read 0x8fb8f6 thru 0x8fd81d (0x1f28 bytes) 000326[0] 1758734005.042243 write 0x9ed9f3 thru 0x9f602a (0x8638 bytes) 000327[0] 1758734005.083716 mapwrite 0x6b5cea thru 0x6ba9ef (0x4d06 bytes) 000329[1] 1758734005.188175 write 0x05fc8e thru 0x062b50 (0x2ec3 bytes) 000330[0] 1758734005.214590 read 0x748307 thru 0x751cfe (0x99f8 bytes) 000331[1] 1758734005.240211 read 0x787054 thru 0x78b457 (0x4404 bytes) 000332[1] 1758734005.264126 mapwrite 0x0748da thru 0x0813c8 (0xcaef bytes) 000334[1] 1758734005.374937 mapwrite 0x2d8dca thru 0x2e7fc2 (0xf1f9 bytes) 000335[1] 1758734005.420167 mapwrite 0x77ab81 thru 0x783e44 (0x92c4 bytes) 000336[0] 1758734005.476166 mapwrite 0x63e5ba thru 0x63ea23 (0x46a bytes) 000338[1] 1758734005.552738 mapread 0x3d5f5e thru 0x3e1583 (0xb626 bytes) 000339[1] 1758734005.599158 mapwrite 0x49ee1e thru 0x4aa267 (0xb44a bytes) 000341[1] 1758734005.685535 write 0x1486dc thru 0x14d487 (0x4dac bytes) 000342[1] 1758734005.715446 mapwrite 0x30c4f2 thru 0x31601b (0x9b2a bytes) 000344[0] 1758734005.806469 trunc from 0x9f602b to 0x322cbd (0x6d336d bytes) 000345[0] 1758734005.859649 read 0x205677 thru 0x214323 (0xecad bytes) 000346[0] 1758734005.903368 read 0x0e1fee thru 0x0f1cba (0xfccd bytes) 000349[1] 1758734005.935591 trunc from 0x322cbe to 0x273c5b (0xaf062 bytes) 000353[1] 1758734006.008595 mapread 0x15c66d thru 0x166a40 (0xa3d4 bytes) 000354[1] 1758734006.021955 read 0x20b8a9 thru 0x218d03 (0xd45b bytes) 000360[1] 1758734006.138991 write 0x52d825 thru 0x53c26e (0xea4a bytes) 000362[0] 1758734006.141478 mapwrite 0x4e57ea thru 0x4eb110 (0x5927 bytes) 000363[1] 1758734006.258828 mapread 0x1a1cee thru 0x1a688a (0x4b9d bytes) 000370[0] 1758734006.370570 trunc from 0x53c26f to 0x36e996 (0x1cd8d8 bytes) 000371[0] 1758734006.422123 write 0x908fa0 thru 0x90c41b (0x347c bytes) 000372[0] 1758734006.434992 write 0x8eb15e thru 0x8ec85b (0x16fe bytes) 000373[1] 1758734006.507970 write 0x8172ee thru 0x820d2c (0x9a3f bytes) 000378[0] 1758734006.620072 write 0x9c8ba6 thru 0x9d47ff (0xbc5a bytes) 000379[1] 1758734006.643461 mapread 0x2eb45c thru 0x2f86ae (0xd253 bytes) 000381[0] 1758734006.773563 mapwrite 0x87f2d3 thru 0x88b394 (0xc0c2 bytes) 000382[1] 1758734006.871467 mapwrite 0x7f6fbf thru 0x801dd8 (0xae1a bytes) 000383[0] 1758734007.184734 read 0x8281c0 thru 0x82aaba (0x28fb bytes) 000385[0] 1758734007.214948 mapread 0x86797e thru 0x8719a2 (0xa025 bytes) 000390[1] 1758734007.262324 read 0x10cca0 thru 0x10cdc4 (0x125 bytes) 000391[1] 1758734007.268831 trunc from 0x9d4800 to 0x422f93 (0x5b186c bytes) 000392[1] 1758734007.317532 write 0x71fd81 thru 0x72a86e (0xaaee bytes) 000393[1] 1758734007.344067 mapread 0x2d003e thru 0x2d8db8 (0x8d7b bytes) 000395[1] 1758734007.381443 write 0x22aa2e thru 0x22cb21 (0x20f4 bytes) 000396[1] 1758734007.398440 mapread 0x13c453 thru 0x147453 (0xb001 bytes) 000397[0] 1758734007.437214 trunc from 0x72a86f to 0x440fbd (0x2e98b1 bytes) 000400[0] 1758734007.520360 mapwrite 0x860be8 thru 0x86958f (0x89a8 bytes) 000403[0] 1758734007.617721 read 0x35c470 thru 0x361003 (0x4b94 bytes) 000404[1] 1758734007.642747 write 0x31ae6b thru 0x326cf6 (0xbe8c bytes) 000405[0] 1758734007.680487 mapwrite 0x7c584f thru 0x7ccad1 (0x7283 bytes) 000407[1] 1758734007.773781 write 0x75b240 thru 0x76380c (0x85cd bytes) 000408[1] 1758734007.791847 write 0x5d88d5 thru 0x5e1f9d (0x96c9 bytes) 000410[1] 1758734007.828155 write 0x70f12c thru 0x71b1bb (0xc090 bytes) 000411[0] 1758734007.849422 mapwrite 0x4f23d1 thru 0x4f7fb2 (0x5be2 bytes) 000412[1] 1758734007.938162 mapread 0x37653e thru 0x377845 (0x1308 bytes) 000414[1] 1758734007.981544 write 0x01a471 thru 0x01fea9 (0x5a39 bytes) 000415[0] 1758734007.999058 mapwrite 0x135a14 thru 0x140648 (0xac35 bytes) 000418[0] 1758734008.095063 mapwrite 0x0fa3dc thru 0x0fddd9 (0x39fe bytes) 000420[1] 1758734008.157184 read 0x354a8e thru 0x358fa8 (0x451b bytes) 000421[0] 1758734008.184427 mapwrite 0x4b5c02 thru 0x4bad1d (0x511c bytes) 000422[0] 1758734008.242407 mapwrite 0x35a48e thru 0x35c7ed (0x2360 bytes) 000423[0] 1758734008.310720 mapwrite 0x6404e5 thru 0x64c9d2 (0xc4ee bytes) 000427[0] 1758734008.399071 trunc from 0x869590 to 0x29b704 (0x5cde8b bytes) 000428[0] 1758734008.412512 mapwrite 0x81beea thru 0x81bf33 (0x4a bytes) 000429[0] 1758734008.488205 mapread 0x0a9888 thru 0x0aefbd (0x5736 bytes) 000430[0] 1758734008.522635 write 0x40c61b thru 0x4124ee (0x5ed4 bytes) 000431[1] 1758734008.543141 mapwrite 0x3fe428 thru 0x402abc (0x4695 bytes) 000432[1] 1758734008.627612 trunc from 0x81bf34 to 0x378f2a (0x4a3009 bytes) 000433[1] 1758734008.639554 write 0x0e06a3 thru 0x0e28fd (0x225b bytes) 000434[1] 1758734008.665485 trunc from 0x378f2b to 0x7fc43b (0x483511 bytes) 000435[1] 1758734008.684090 read 0x38d47c thru 0x39c7db (0xf360 bytes) 000436[1] 1758734008.708109 read 0x652d32 thru 0x654f54 (0x2223 bytes) 000438[0] 1758734008.713189 mapread 0x30df9a thru 0x317051 (0x90b8 bytes) 000439[1] 1758734008.784171 read 0x110fae thru 0x11e506 (0xd559 bytes) 000442[0] 1758734008.832709 read 0x217770 thru 0x218c4b (0x14dc bytes) 000444[1] 1758734008.853718 mapwrite 0x184353 thru 0x18d483 (0x9131 bytes) 000447[0] 1758734008.968461 mapread 0x07c815 thru 0x08bd36 (0xf522 bytes) 000448[0] 1758734009.028734 mapwrite 0x5b7931 thru 0x5b7d88 (0x458 bytes) 000450[0] 1758734009.117081 write 0x4f76ef thru 0x4fbab2 (0x43c4 bytes) 000451[0] 1758734009.128495 read 0x1cf2c5 thru 0x1d3862 (0x459e bytes) 000453[1] 1758734009.151553 trunc from 0x7fc43c to 0x24561e (0x5b6e1d bytes) 000460[0] 1758734009.250348 read 0x017dc5 thru 0x024f31 (0xd16d bytes) 000461[0] 1758734009.312649 read 0x185b59 thru 0x18e51f (0x89c7 bytes) 000462[1] 1758734009.317573 read 0x192513 thru 0x193818 (0x1306 bytes) 000465[0] 1758734009.373947 mapread 0x21758f thru 0x21e45e (0x6ed0 bytes) 000466[0] 1758734009.403015 mapread 0x11e363 thru 0x12d995 (0xf633 bytes) 000467[0] 1758734009.420997 write 0x1ceb4a thru 0x1da3ed (0xb8a4 bytes) 000469[1] 1758734009.490135 mapread 0x0e8050 thru 0x0f7339 (0xf2ea bytes) 000470[1] 1758734009.603243 read 0x18d286 thru 0x19be86 (0xec01 bytes) 000471[0] 1758734009.648569 trunc from 0x24561f to 0x39cce4 (0x1576c6 bytes) 000473[1] 1758734009.710735 mapwrite 0x8da0a7 thru 0x8e5868 (0xb7c2 bytes) 000474[0] 1758734009.901761 read 0x5fe733 thru 0x606a3b (0x8309 bytes) 000480[1] 1758734010.024231 mapwrite 0x1a287e thru 0x1a803b (0x57be bytes) 000485[1] 1758734010.146084 read 0x666b4f thru 0x66a851 (0x3d03 bytes) 000487[1] 1758734010.165042 write 0x5ed233 thru 0x5f855b (0xb329 bytes) 000492[1] 1758734010.238338 mapread 0x2b8065 thru 0x2c3aeb (0xba87 bytes) 000495[0] 1758734010.271011 trunc from 0x8e5869 to 0x68ae70 (0x25a9f8 bytes) 000498[1] 1758734010.362451 trunc from 0x68ae71 to 0x3d3112 (0x2b7d5e bytes) 000499[0] 1758734010.403496 write 0x0354f8 thru 0x03d1af (0x7cb8 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4010 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1758734011.039481 write 0x9c5000 thru 0x9c7c9f (0x2ca0 bytes) truncating to largest ever: 0x9975f7 000008[1] 1758734011.079070 trunc from 0x9c7ca0 to 0x9975f6 (0x306a9 bytes) 000009[1] 1758734011.095326 write 0x641000 thru 0x6481f4 (0x71f5 bytes) 000013[1] 1758734011.121245 mapread 0x07c000 thru 0x089879 (0xd87a bytes) 000017[0] 1758734011.162293 read 0x4b1000 thru 0x4b33cf (0x23d0 bytes) 000018[1] 1758734011.188812 write 0x485000 thru 0x489dcd (0x4dce bytes) 000019[0] 1758734011.204125 trunc from 0x9975f7 to 0x8bf7c6 (0xd7e30 bytes) 000020[1] 1758734011.264325 read 0x285000 thru 0x28d462 (0x8463 bytes) 000025[0] 1758734011.287330 mapwrite 0x293000 thru 0x2a09c1 (0xd9c2 bytes) 000026[0] 1758734011.375326 trunc from 0x8bf7c7 to 0x5003d9 (0x3bf3ed bytes) 000032[0] 1758734011.446525 read 0x3b3000 thru 0x3b6946 (0x3947 bytes) 000037[1] 1758734011.465499 read 0x42d000 thru 0x432f74 (0x5f75 bytes) 000038[0] 1758734011.480973 mapwrite 0x5d8000 thru 0x5de6b6 (0x66b7 bytes) 000041[0] 1758734011.572049 write 0x097000 thru 0x099c55 (0x2c56 bytes) 000042[1] 1758734011.586093 read 0x39a000 thru 0x39a87a (0x87b bytes) 000044[0] 1758734011.608603 mapwrite 0x041000 thru 0x046e4e (0x5e4f bytes) 000046[1] 1758734011.663660 mapread 0x08d000 thru 0x0944bb (0x74bc bytes) 000050[0] 1758734011.705518 trunc from 0x5de6b7 to 0x5beaf0 (0x1fbc6 bytes) 000052[1] 1758734011.738204 mapwrite 0x0c7000 thru 0x0d0ee7 (0x9ee8 bytes) 000053[0] 1758734011.830369 read 0x0be000 thru 0x0c252b (0x452c bytes) 000054[1] 1758734011.851172 mapwrite 0x3d3000 thru 0x3de535 (0xb536 bytes) 000055[1] 1758734011.932517 trunc from 0x5beaf1 to 0x87101e (0x2b252e bytes) 000056[1] 1758734011.961036 mapwrite 0x30c000 thru 0x317d98 (0xbd99 bytes) 000058[1] 1758734012.016556 mapwrite 0x606000 thru 0x60935e (0x335f bytes) 000059[1] 1758734012.118432 read 0x176000 thru 0x179c7a (0x3c7b bytes) 000060[0] 1758734012.140455 write 0x8b4000 thru 0x8ba7bb (0x67bc bytes) 000063[1] 1758734012.156768 write 0x324000 thru 0x3282fc (0x42fd bytes) 000064[1] 1758734012.177289 trunc from 0x8ba7bc to 0x1b3b26 (0x706c95 bytes) 000066[1] 1758734012.216279 write 0x497000 thru 0x49836a (0x136b bytes) 000069[0] 1758734012.218218 write 0x0fa000 thru 0x1054da (0xb4db bytes) 000070[1] 1758734012.242488 read 0x1a1000 thru 0x1acc7b (0xbc7c bytes) 000073[0] 1758734012.262762 trunc from 0x49836b to 0x0c6a1b (0x3d194f bytes) 000074[0] 1758734012.285458 mapread 0x029000 thru 0x037ada (0xeadb bytes) 000075[0] 1758734012.300098 mapread 0x082000 thru 0x083e41 (0x1e42 bytes) 000082[0] 1758734012.358243 read 0x005000 thru 0x012f13 (0xdf14 bytes) 000084[0] 1758734012.384253 mapwrite 0x201000 thru 0x207643 (0x6644 bytes) 000087[0] 1758734012.523220 mapread 0x0ec000 thru 0x0f1184 (0x5185 bytes) 000089[1] 1758734012.540021 mapwrite 0x42e000 thru 0x4310fe (0x30ff bytes) 000090[0] 1758734012.656494 read 0x22d000 thru 0x23ba45 (0xea46 bytes) 000091[0] 1758734012.673624 mapread 0x26d000 thru 0x275527 (0x8528 bytes) 000093[0] 1758734012.711472 mapread 0x149000 thru 0x14ed41 (0x5d42 bytes) 000094[1] 1758734012.736548 mapwrite 0x899000 thru 0x8a170f (0x8710 bytes) 000095[0] 1758734012.887558 trunc from 0x8a1710 to 0x64018f (0x261580 bytes) 000096[1] 1758734012.934805 mapwrite 0x180000 thru 0x18586d (0x586e bytes) 000099[1] 1758734013.048581 mapread 0x5fa000 thru 0x6014d5 (0x74d6 bytes) 000100[1] 1758734013.077785 read 0x405000 thru 0x40a02c (0x502d bytes) 000101[0] 1758734013.088374 read 0x4c0000 thru 0x4cf022 (0xf023 bytes) 000103[1] 1758734013.110186 write 0x0eb000 thru 0x0f099f (0x59a0 bytes) 000104[1] 1758734013.125442 write 0x2fc000 thru 0x3050a9 (0x90aa bytes) 000105[1] 1758734013.130205 write 0x92e000 thru 0x93a91d (0xc91e bytes) 000106[0] 1758734013.154542 read 0x187000 thru 0x192ee6 (0xbee7 bytes) 000108[0] 1758734013.200748 read 0x8eb000 thru 0x8fa8cc (0xf8cd bytes) 000109[0] 1758734013.213639 mapread 0x365000 thru 0x367107 (0x2108 bytes) 000110[1] 1758734013.237422 read 0x12e000 thru 0x1344e2 (0x64e3 bytes) 000111[1] 1758734013.248456 write 0x3d2000 thru 0x3d540f (0x3410 bytes) 000113[1] 1758734013.264695 mapread 0x501000 thru 0x506179 (0x517a bytes) 000118[0] 1758734013.336655 read 0x1d8000 thru 0x1e2c35 (0xac36 bytes) 000121[0] 1758734013.370157 mapwrite 0x84f000 thru 0x85857f (0x9580 bytes) 000122[0] 1758734013.440041 trunc from 0x93a91e to 0x6d07bd (0x26a160 bytes) 000123[1] 1758734013.482681 mapread 0x599000 thru 0x59df56 (0x4f57 bytes) 000125[0] 1758734013.513681 mapwrite 0x638000 thru 0x6420b3 (0xa0b4 bytes) 000126[0] 1758734013.610852 write 0x6a1000 thru 0x6a4f41 (0x3f42 bytes) 000127[0] 1758734013.623302 mapread 0x1ff000 thru 0x200d0d (0x1d0e bytes) 000128[1] 1758734013.632376 mapwrite 0x063000 thru 0x067c7b (0x4c7c bytes) 000129[1] 1758734013.718340 write 0x213000 thru 0x221924 (0xe925 bytes) 000131[1] 1758734013.723782 read 0x022000 thru 0x02eb08 (0xcb09 bytes) 000132[0] 1758734013.736013 mapwrite 0x9f8000 thru 0x9ff6d8 (0x76d9 bytes) 000133[0] 1758734013.852701 write 0x8b3000 thru 0x8bca52 (0x9a53 bytes) 000134[0] 1758734013.873519 mapwrite 0x381000 thru 0x3832fa (0x22fb bytes) 000135[0] 1758734013.931792 trunc from 0x9ff6d9 to 0x148b14 (0x8b6bc4 bytes) 000136[1] 1758734013.970975 read 0x05f000 thru 0x062a65 (0x3a66 bytes) 000140[0] 1758734014.025792 mapread 0x054000 thru 0x056b8c (0x2b8d bytes) 000142[0] 1758734014.041042 write 0x00e000 thru 0x01babc (0xdabd bytes) 000143[1] 1758734014.089083 read 0x085000 thru 0x086be4 (0x1be5 bytes) 000147[0] 1758734014.134358 mapwrite 0x306000 thru 0x314063 (0xe064 bytes) 000149[0] 1758734014.245856 mapwrite 0x5db000 thru 0x5e6ec5 (0xbec6 bytes) 000150[0] 1758734014.338064 mapread 0x412000 thru 0x41cacb (0xaacc bytes) 000151[1] 1758734014.363218 trunc from 0x5e6ec6 to 0x6d5371 (0xee4ac bytes) 000153[0] 1758734014.378589 read 0x21e000 thru 0x229a05 (0xba06 bytes) 000154[0] 1758734014.392685 write 0x005000 thru 0x005d8a (0xd8b bytes) truncating to largest ever: 0x9bc03b 000155[0] 1758734014.404973 trunc from 0x6d5372 to 0x9bc03a (0x2e6cc9 bytes) 000158[0] 1758734014.463166 mapread 0x7a7000 thru 0x7ad28c (0x628d bytes) 000159[0] 1758734014.494975 mapwrite 0x875000 thru 0x879c3e (0x4c3f bytes) 000160[0] 1758734014.542155 mapwrite 0x7dd000 thru 0x7e743f (0xa440 bytes) 000161[0] 1758734014.591114 mapwrite 0x712000 thru 0x721dae (0xfdaf bytes) 000162[0] 1758734014.656376 mapwrite 0x996000 thru 0x9a4811 (0xe812 bytes) 000163[1] 1758734014.717701 read 0x504000 thru 0x5093a4 (0x53a5 bytes) 000164[0] 1758734014.741062 mapwrite 0x3a5000 thru 0x3ad4b7 (0x84b8 bytes) 000165[0] 1758734014.810103 write 0x7ca000 thru 0x7d0055 (0x6056 bytes) 000166[1] 1758734014.824984 mapwrite 0x567000 thru 0x574c93 (0xdc94 bytes) 000167[0] 1758734014.935788 write 0x561000 thru 0x569832 (0x8833 bytes) 000168[1] 1758734014.961970 trunc from 0x9bc03b to 0x45fc87 (0x55c3b3 bytes) 000172[0] 1758734015.027287 write 0x8f9000 thru 0x907b28 (0xeb29 bytes) 000173[1] 1758734015.042062 write 0x6e6000 thru 0x6f2e9f (0xcea0 bytes) 000178[1] 1758734015.084268 write 0x07f000 thru 0x08db04 (0xeb05 bytes) 000180[0] 1758734015.100760 trunc from 0x907b29 to 0x1e3bb1 (0x723f77 bytes) 000182[0] 1758734015.170729 mapread 0x117000 thru 0x126559 (0xf55a bytes) 000185[1] 1758734015.224800 mapwrite 0x24f000 thru 0x254687 (0x5688 bytes) 000186[0] 1758734015.340593 write 0x6ea000 thru 0x6f2fc9 (0x8fca bytes) 000187[0] 1758734015.350794 mapwrite 0x49f000 thru 0x49fd22 (0xd23 bytes) 000190[1] 1758734015.406450 write 0x51c000 thru 0x5259dc (0x99dd bytes) 000192[0] 1758734015.443738 mapread 0x2f3000 thru 0x2f529d (0x229e bytes) 000193[0] 1758734015.479129 trunc from 0x6f2fca to 0x520854 (0x1d2775 bytes) 000197[0] 1758734015.530224 trunc from 0x520855 to 0x922287 (0x401a33 bytes) 000198[0] 1758734015.576530 mapread 0x75d000 thru 0x7639e8 (0x69e9 bytes) 000200[1] 1758734015.617005 read 0x679000 thru 0x6813cf (0x83d0 bytes) 000201[1] 1758734015.634797 read 0x25d000 thru 0x261197 (0x4198 bytes) 000202[1] 1758734015.645026 read 0x3d2000 thru 0x3d9f19 (0x7f1a bytes) 000203[1] 1758734015.652691 mapwrite 0x94d000 thru 0x95645e (0x945f bytes) 000204[0] 1758734015.744507 read 0x931000 thru 0x938f68 (0x7f69 bytes) 000205[1] 1758734015.764954 read 0x6c0000 thru 0x6cf032 (0xf033 bytes) truncating to largest ever: 0x9c4906 000206[0] 1758734015.778309 trunc from 0x95645f to 0x9c4905 (0x6e4a7 bytes) 000214[1] 1758734015.899928 read 0x12a000 thru 0x12ea5a (0x4a5b bytes) 000215[0] 1758734015.912436 write 0x024000 thru 0x02b020 (0x7021 bytes) 000216[0] 1758734015.928575 mapread 0x10a000 thru 0x10d9c4 (0x39c5 bytes) 000218[0] 1758734015.939218 write 0x801000 thru 0x80b85a (0xa85b bytes) 000219[0] 1758734015.945786 read 0x5e9000 thru 0x5eb5c7 (0x25c8 bytes) 000220[1] 1758734015.950623 read 0x0b2000 thru 0x0bef43 (0xcf44 bytes) 000222[1] 1758734015.976439 write 0x392000 thru 0x39cb84 (0xab85 bytes) 000223[1] 1758734016.001630 mapwrite 0x8ba000 thru 0x8c5991 (0xb992 bytes) 000224[0] 1758734016.065359 mapwrite 0x2aa000 thru 0x2abf37 (0x1f38 bytes) 000225[1] 1758734016.138455 mapwrite 0x306000 thru 0x309be4 (0x3be5 bytes) 000226[0] 1758734016.204728 mapwrite 0x84a000 thru 0x851755 (0x7756 bytes) 000230[0] 1758734016.316913 read 0x3ba000 thru 0x3c722c (0xd22d bytes) 000232[1] 1758734016.325814 trunc from 0x9c4906 to 0x0effaa (0x8d495b bytes) 000236[0] 1758734016.368537 mapwrite 0x665000 thru 0x666d72 (0x1d73 bytes) 000237[1] 1758734016.454369 read 0x3a7000 thru 0x3a8f29 (0x1f2a bytes) 000238[1] 1758734016.551354 mapread 0x544000 thru 0x54918a (0x518b bytes) 000239[0] 1758734016.587171 mapwrite 0x21c000 thru 0x228381 (0xc382 bytes) 000244[0] 1758734016.717485 write 0x842000 thru 0x84c93b (0xa93c bytes) 000245[0] 1758734016.733566 write 0x210000 thru 0x2143e2 (0x43e3 bytes) 000246[0] 1758734016.756943 trunc from 0x84c93c to 0x72aa35 (0x121f06 bytes) 000249[0] 1758734016.823274 mapread 0x4fb000 thru 0x4fb4be (0x4bf bytes) 000250[1] 1758734016.830021 mapread 0x612000 thru 0x618350 (0x6351 bytes) 000262[1] 1758734016.872078 write_OD 0x9ab000 thru 0x9acfff (0x2000 bytes) 000267[1] 1758734016.919934 write_OD 0x840000 thru 0x847fff (0x8000 bytes) 000268[0] 1758734016.965160 write 0x0b9000 thru 0x0c3a6f (0xaa70 bytes) 000269[0] 1758734016.996407 mapwrite 0x64a000 thru 0x64e0cf (0x40d0 bytes) 000271[0] 1758734017.100342 trunc from 0x9ad000 to 0x17f24c (0x82ddb3 bytes) 000272[1] 1758734017.123021 read_OD 0x0b8000 thru 0x0bafff (0x3000 bytes) 000273[0] 1758734017.150536 mapwrite 0x4aa000 thru 0x4ac5c6 (0x25c7 bytes) 000275[0] 1758734017.235546 read 0x140000 thru 0x146031 (0x6032 bytes) 000276[1] 1758734017.247789 mapread 0x3e9000 thru 0x3eca07 (0x3a08 bytes) 000277[0] 1758734017.295665 mapread 0x435000 thru 0x4442ed (0xf2ee bytes) 000278[0] 1758734017.322232 write 0x1c9000 thru 0x1d0cca (0x7ccb bytes) 000288[1] 1758734017.433716 read_OD 0x203000 thru 0x203fff (0x1000 bytes) 000289[1] 1758734017.460029 mapread 0x198000 thru 0x19b01b (0x301c bytes) 000290[1] 1758734017.479281 read_OD 0x3ac000 thru 0x3b5fff (0xa000 bytes) 000291[1] 1758734017.485977 write_OD 0x310000 thru 0x317fff (0x8000 bytes) 000292[0] 1758734017.528653 mapwrite 0x4a0000 thru 0x4a83f3 (0x83f4 bytes) 000293[0] 1758734017.607745 read 0x463000 thru 0x46dc75 (0xac76 bytes) 000297[0] 1758734017.654410 write 0x13e000 thru 0x14cf39 (0xef3a bytes) 000302[0] 1758734017.725648 read 0x1a3000 thru 0x1a8257 (0x5258 bytes) 000303[0] 1758734017.731735 trunc from 0x4ac5c7 to 0x3696e5 (0x142ee1 bytes) 000304[1] 1758734017.756943 read_OD 0x102000 thru 0x105fff (0x4000 bytes) 000305[0] 1758734017.793685 trunc from 0x3696e6 to 0x8360ef (0x4cca0a bytes) 000308[1] 1758734017.860978 mapread 0x12c000 thru 0x13b719 (0xf71a bytes) 000312[0] 1758734017.950827 mapwrite 0x209000 thru 0x20b95c (0x295d bytes) 000317[1] 1758734018.022990 mapread 0x495000 thru 0x49ec55 (0x9c56 bytes) 000319[0] 1758734018.088241 read 0x08e000 thru 0x09a05f (0xc060 bytes) 000320[0] 1758734018.102540 mapread 0x1e5000 thru 0x1ed1c0 (0x81c1 bytes) 000324[0] 1758734018.126577 write 0x5aa000 thru 0x5ab634 (0x1635 bytes) 000326[0] 1758734018.153620 read 0x5ad000 thru 0x5b339d (0x639e bytes) 000329[0] 1758734018.186365 read 0x5e1000 thru 0x5e70de (0x60df bytes) 000332[0] 1758734018.218083 mapread 0x29f000 thru 0x2a737b (0x837c bytes) 000338[1] 1758734018.282679 read_OD 0x7dd000 thru 0x7ddfff (0x1000 bytes) 000344[1] 1758734018.315254 mapread 0x0de000 thru 0x0e2207 (0x4208 bytes) 000345[1] 1758734018.334373 mapwrite 0x92b000 thru 0x92cc53 (0x1c54 bytes) 000349[1] 1758734018.440964 read_OD 0x4ba000 thru 0x4bafff (0x1000 bytes) 000350[0] 1758734018.454588 write 0x094000 thru 0x0a0d8a (0xcd8b bytes) 000352[0] 1758734018.487358 trunc from 0x92cc54 to 0x2ee38b (0x63e8c8 bytes) 000353[1] 1758734018.506265 mapread 0x070000 thru 0x073f77 (0x3f78 bytes) 000356[0] 1758734018.574609 mapwrite 0x776000 thru 0x778ec7 (0x2ec8 bytes) 000358[0] 1758734018.663677 read 0x23e000 thru 0x240bb0 (0x2bb1 bytes) 000360[0] 1758734018.672503 read 0x2c9000 thru 0x2d3012 (0xa013 bytes) 000361[0] 1758734018.691540 write 0x7fe000 thru 0x8095c6 (0xb5c7 bytes) 000363[1] 1758734018.724582 mapread 0x276000 thru 0x27a731 (0x4732 bytes) 000370[0] 1758734018.761452 mapread 0x782000 thru 0x786ed0 (0x4ed1 bytes) 000371[0] 1758734018.785415 mapread 0x50a000 thru 0x50ff38 (0x5f39 bytes) 000376[0] 1758734018.800944 write 0x0bd000 thru 0x0c2666 (0x5667 bytes) 000378[0] 1758734018.824124 write 0x8d8000 thru 0x8ded74 (0x6d75 bytes) 000381[1] 1758734018.850010 mapread 0x5d3000 thru 0x5da4c2 (0x74c3 bytes) 000382[0] 1758734018.890649 mapwrite 0x495000 thru 0x49816d (0x316e bytes) 000386[1] 1758734018.991484 mapwrite 0x9ae000 thru 0x9baca7 (0xcca8 bytes) 000389[0] 1758734019.141059 trunc from 0x9baca8 to 0x7c8f92 (0x1f1d15 bytes) 000393[0] 1758734019.200970 write 0x6a0000 thru 0x6aa5fb (0xa5fc bytes) 000396[1] 1758734019.243723 trunc from 0x7c8f93 to 0x1213df (0x6a7bb3 bytes) 000397[0] 1758734019.288477 read 0x0e5000 thru 0x0ecad0 (0x7ad1 bytes) 000399[0] 1758734019.313607 mapwrite 0x9b7000 thru 0x9c0929 (0x992a bytes) 000400[1] 1758734019.448468 trunc from 0x9c092a to 0x65d8eb (0x36303e bytes) 000402[1] 1758734019.475616 mapwrite 0x6c4000 thru 0x6cdde2 (0x9de3 bytes) 000403[0] 1758734019.556936 mapread 0x6b4000 thru 0x6b8d53 (0x4d54 bytes) 000404[0] 1758734019.596591 read 0x1cf000 thru 0x1d687f (0x7880 bytes) 000406[0] 1758734019.609410 mapwrite 0x020000 thru 0x02deff (0xdf00 bytes) 000407[1] 1758734019.701240 trunc from 0x6cdde3 to 0x377b2a (0x3562b8 bytes) 000410[0] 1758734019.767838 mapread 0x286000 thru 0x28e1f4 (0x81f5 bytes) 000411[0] 1758734019.803173 read 0x2ac000 thru 0x2b435a (0x835b bytes) 000412[1] 1758734019.859505 read_OD 0x163000 thru 0x171fff (0xf000 bytes) 000413[0] 1758734019.868612 read 0x1f3000 thru 0x1f7d8a (0x4d8b bytes) 000416[1] 1758734019.869379 write_OD 0x06b000 thru 0x074fff (0xa000 bytes) 000419[1] 1758734019.925239 write_OD 0x786000 thru 0x787fff (0x2000 bytes) 000421[0] 1758734019.977010 mapread 0x62a000 thru 0x63271b (0x871c bytes) 000422[0] 1758734020.080017 trunc from 0x788000 to 0x82b14a (0xa314b bytes) 000423[1] 1758734020.136706 read_OD 0x524000 thru 0x526fff (0x3000 bytes) 000424[1] 1758734020.165032 read_OD 0x1de000 thru 0x1e9fff (0xc000 bytes) 000426[1] 1758734020.175545 mapread 0x336000 thru 0x33ceb1 (0x6eb2 bytes) 000427[0] 1758734020.191677 mapwrite 0x5d3000 thru 0x5e035a (0xd35b bytes) 000431[1] 1758734020.326970 mapwrite 0x01f000 thru 0x023eda (0x4edb bytes) 000432[0] 1758734020.393645 mapread 0x30b000 thru 0x30c692 (0x1693 bytes) 000434[1] 1758734020.423429 trunc from 0x82b14b to 0x0f7c15 (0x733535 bytes) 000437[0] 1758734020.477859 mapwrite 0x988000 thru 0x9900e0 (0x80e1 bytes) 000441[0] 1758734020.601493 read 0x82e000 thru 0x8348cc (0x68cd bytes) 000442[1] 1758734020.621839 write_OD 0x3fe000 thru 0x408fff (0xb000 bytes) 000443[1] 1758734020.676192 write_OD 0x60e000 thru 0x617fff (0xa000 bytes) 000445[1] 1758734020.728358 write_OD 0x3c6000 thru 0x3c9fff (0x4000 bytes) 000449[0] 1758734020.776400 write 0x150000 thru 0x15059b (0x59c bytes) 000450[1] 1758734020.812101 write_OD 0x302000 thru 0x30afff (0x9000 bytes) 000451[1] 1758734020.878229 mapwrite 0x605000 thru 0x612af7 (0xdaf8 bytes) 000452[0] 1758734020.968018 trunc from 0x9900e1 to 0x78ecf3 (0x2013ed bytes) 000453[1] 1758734021.011038 mapwrite 0x686000 thru 0x695f76 (0xff77 bytes) 000454[1] 1758734021.142083 trunc from 0x78ecf4 to 0x530d74 (0x25df7f bytes) 000456[0] 1758734021.209710 mapwrite 0x5c6000 thru 0x5d5221 (0xf222 bytes) 000457[1] 1758734021.328255 read_OD 0x1a1000 thru 0x1adfff (0xd000 bytes) 000461[0] 1758734021.358420 mapread 0x3f6000 thru 0x3f7cd5 (0x1cd6 bytes) 000463[0] 1758734021.365002 trunc from 0x5d5222 to 0x718894 (0x143673 bytes) 000464[0] 1758734021.393189 mapwrite 0x6e7000 thru 0x6eef1f (0x7f20 bytes) 000465[0] 1758734021.460241 read 0x241000 thru 0x249ced (0x8cee bytes) 000466[0] 1758734021.474677 write 0x1cc000 thru 0x1d8dfb (0xcdfc bytes) 000469[0] 1758734021.515311 trunc from 0x718895 to 0x5e6104 (0x132790 bytes) 000470[1] 1758734021.537681 mapwrite 0x90b000 thru 0x90fbd0 (0x4bd1 bytes) 000473[0] 1758734021.694655 trunc from 0x90fbd1 to 0x84ed06 (0xc0eca bytes) 000476[1] 1758734021.767771 mapread 0x180000 thru 0x181afc (0x1afd bytes) 000477[1] 1758734021.813958 mapread 0x07c000 thru 0x087593 (0xb594 bytes) 000479[1] 1758734021.849535 mapwrite 0x465000 thru 0x46f1d1 (0xa1d2 bytes) 000481[1] 1758734021.956133 read_OD 0x3a4000 thru 0x3affff (0xc000 bytes) 000484[1] 1758734021.981562 write_OD 0x823000 thru 0x829fff (0x7000 bytes) 000486[0] 1758734022.019489 mapread 0x520000 thru 0x52412d (0x412e bytes) 000488[1] 1758734022.042043 trunc from 0x84ed07 to 0x328131 (0x526bd5 bytes) 000490[1] 1758734022.090494 mapread 0x29b000 thru 0x2a8e86 (0xde87 bytes) 000491[1] 1758734022.109273 trunc from 0x328132 to 0x8a0a45 (0x578914 bytes) 000492[1] 1758734022.130339 mapwrite 0x327000 thru 0x333bd9 (0xcbda bytes) 000494[0] 1758734022.200637 mapwrite 0x740000 thru 0x74b85b (0xb85c bytes) 000497[0] 1758734022.295717 mapwrite 0x73d000 thru 0x7407b3 (0x37b4 bytes) 000499[1] 1758734022.328672 write_OD 0x3bf000 thru 0x3c1fff (0x3000 bytes) All operations completed A-OK! PASS 16a (34s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 13:13:45 (1758734025) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.387208 s, 21.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.244069 s, 34.4 MB/s Chance of close/open is 1 in 50 Seed set to 4027 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x492c4e 000002[1] 1758734027.844986 trunc from 00000000 to 0x492c4d (0x492c4e bytes) 000011[0] 1758734028.009727 mapwrite 0x63a1b3 thru 0x63a5ae (0x3fc bytes) 000012[1] 1758734028.093488 read 0x63592e thru 0x63599b (0x6e bytes) 000013[0] 1758734028.114000 mapwrite 0x260d61 thru 0x261c1d (0xebd bytes) 000014[0] 1758734028.172758 write 0x9bab4f thru 0x9bbf96 (0x1448 bytes) 000015[1] 1758734028.174566 write 0x224552 thru 0x224afa (0x5a9 bytes) 000016[0] 1758734028.198047 mapwrite 0x14bcf3 thru 0x14d31a (0x1628 bytes) truncating to largest ever: 0x4ced3a 000017[1] 1758734028.330957 trunc from 0x9bbf97 to 0x4ced39 (0x4ed25d bytes) 000018[0] 1758734028.391268 read 0x4bd4cd thru 0x4bd829 (0x35d bytes) 000021[0] 1758734028.455323 mapwrite 0x583480 thru 0x5835fc (0x17d bytes) 000022[1] 1758734028.631481 mapread 0x452286 thru 0x453e02 (0x1b7d bytes) 000025[1] 1758734028.694967 mapwrite 0x01cc0d thru 0x01ebbb (0x1faf bytes) 000027[1] 1758734028.785707 read 0x1928e6 thru 0x193077 (0x792 bytes) 000028[1] 1758734028.790379 mapwrite 0x92a3ce thru 0x92b7ec (0x141f bytes) truncating to largest ever: 0x7783c9 000029[1] 1758734028.883196 trunc from 0x92b7ed to 0x7783c8 (0x1b3424 bytes) 000030[1] 1758734028.905443 write 0x2c5899 thru 0x2c5edd (0x645 bytes) 000031[1] 1758734028.937448 mapwrite 0x804674 thru 0x805ead (0x183a bytes) 000032[0] 1758734029.006704 mapread 0x289b8c thru 0x289bc7 (0x3c bytes) 000034[1] 1758734029.042036 read 0x2d5e7d thru 0x2d78e4 (0x1a68 bytes) 000036[1] 1758734029.047616 mapread 0x132391 thru 0x13239d (0xd bytes) 000037[0] 1758734029.075195 mapwrite 0x58fdeb thru 0x5903b7 (0x5cd bytes) 000040[0] 1758734029.184185 write 0x87262e thru 0x8738fa (0x12cd bytes) 000042[1] 1758734029.198176 write 0x396590 thru 0x397f31 (0x19a2 bytes) 000043[1] 1758734029.256029 write 0x232eb4 thru 0x2339a4 (0xaf1 bytes) 000046[0] 1758734029.312081 trunc from 0x8738fb to 0x385ffa (0x4ed900 bytes) 000047[0] 1758734029.346454 mapread 0x187931 thru 0x1882b8 (0x988 bytes) 000048[0] 1758734029.354480 mapwrite 0x14a068 thru 0x14a443 (0x3dc bytes) 000049[1] 1758734029.429053 mapread 0x1bf3c7 thru 0x1bff57 (0xb91 bytes) 000050[0] 1758734029.469697 mapwrite 0x705a99 thru 0x7064e7 (0xa4f bytes) 000051[1] 1758734029.544916 read 0x59a9c4 thru 0x59ae9a (0x4d7 bytes) 000053[0] 1758734029.597126 trunc from 0x7064e8 to 0x1876b1 (0x57ee36 bytes) 000057[1] 1758734029.661110 read 0x0ce9f4 thru 0x0cfc77 (0x1284 bytes) 000059[1] 1758734029.715868 mapwrite 0x1a05e8 thru 0x1a17b0 (0x11c9 bytes) 000065[1] 1758734029.881677 trunc from 0x1a17b1 to 0x5ed981 (0x44c1d1 bytes) 000066[1] 1758734029.914219 write 0x5c25f3 thru 0x5c3414 (0xe22 bytes) 000068[0] 1758734029.991394 mapwrite 0x482e17 thru 0x48395e (0xb48 bytes) 000069[1] 1758734030.105744 trunc from 0x5ed982 to 0x73e4ba (0x150b39 bytes) 000070[0] 1758734030.134827 mapread 0x17ada6 thru 0x17b84c (0xaa7 bytes) 000071[1] 1758734030.177392 trunc from 0x73e4bb to 0x397aae (0x3a6a0c bytes) truncating to largest ever: 0x8e94ad 000072[0] 1758734030.204412 trunc from 0x397aaf to 0x8e94ac (0x5519fe bytes) 000075[0] 1758734030.289263 mapwrite 0x62abca thru 0x62c1b8 (0x15ef bytes) 000076[1] 1758734030.377180 mapread 0x3f1f5b thru 0x3f2e8e (0xf34 bytes) 000078[0] 1758734030.431634 trunc from 0x8e94ad to 0x736efe (0x1b25ae bytes) 000079[1] 1758734030.467024 trunc from 0x736eff to 0x0bdc39 (0x6792c5 bytes) 000080[0] 1758734030.515295 mapread 0x034519 thru 0x034df4 (0x8dc bytes) 000082[0] 1758734030.546556 read 0x032024 thru 0x03230a (0x2e7 bytes) 000083[0] 1758734030.584621 write 0x5f21ad thru 0x5f3c8a (0x1ade bytes) 000090[1] 1758734030.638411 mapread 0x18dba8 thru 0x18f147 (0x15a0 bytes) 000093[1] 1758734030.671522 trunc from 0x5f3c8b to 0x5bf410 (0x3487a bytes) 000095[1] 1758734030.719296 mapread 0x095f62 thru 0x096c4a (0xce9 bytes) 000097[1] 1758734030.749493 read 0x536175 thru 0x5368d2 (0x75e bytes) 000098[0] 1758734030.756288 read 0x0390ef thru 0x03ac23 (0x1b35 bytes) 000100[0] 1758734030.797729 read 0x0b3092 thru 0x0b4b54 (0x1ac3 bytes) truncating to largest ever: 0x939b09 000105[1] 1758734030.873447 trunc from 0x5bf411 to 0x939b08 (0x37a6f8 bytes) 000106[1] 1758734030.902501 mapwrite 0x343583 thru 0x344017 (0xa95 bytes) 000111[1] 1758734031.002429 mapread 0x00432c thru 0x004eb0 (0xb85 bytes) 000113[0] 1758734031.038747 read 0x068829 thru 0x068e8f (0x667 bytes) 000114[1] 1758734031.064333 trunc from 0x939b09 to 0x3498e8 (0x5f0220 bytes) 000115[0] 1758734031.090750 write 0x4e32c0 thru 0x4e3f4c (0xc8d bytes) 000116[1] 1758734031.135245 mapwrite 0x2b7b51 thru 0x2b96aa (0x1b5a bytes) 000120[0] 1758734031.296682 mapwrite 0x998a5a thru 0x99a88d (0x1e34 bytes) 000121[0] 1758734031.440012 mapwrite 0x67b7f3 thru 0x67c41a (0xc28 bytes) 000122[0] 1758734031.495339 trunc from 0x99a88e to 0x4262e1 (0x5745ac bytes) 000124[0] 1758734031.531314 write 0x0cb631 thru 0x0cc95d (0x132d bytes) 000128[0] 1758734031.568309 mapread 0x2b9ae5 thru 0x2b9ed6 (0x3f2 bytes) 000132[0] 1758734031.606747 write 0x425b00 thru 0x425e16 (0x317 bytes) 000133[1] 1758734031.612927 read 0x3b11d5 thru 0x3b28bb (0x16e7 bytes) 000136[1] 1758734031.647490 write 0x716ae7 thru 0x717f91 (0x14ab bytes) 000140[0] 1758734031.687125 mapwrite 0x4d74e8 thru 0x4d8609 (0x1122 bytes) 000142[0] 1758734031.786280 mapwrite 0x04ecef thru 0x04f41f (0x731 bytes) 000143[0] 1758734031.902260 mapread 0x2b3f80 thru 0x2b4665 (0x6e6 bytes) 000146[0] 1758734031.988202 mapwrite 0x9dfeda thru 0x9e15c7 (0x16ee bytes) 000150[1] 1758734032.073744 write 0x5e1cc0 thru 0x5e3a51 (0x1d92 bytes) 000152[1] 1758734032.120738 read 0x4e8d3c thru 0x4e9839 (0xafe bytes) 000153[0] 1758734032.138809 read 0x6f54b8 thru 0x6f7041 (0x1b8a bytes) 000154[0] 1758734032.163081 mapwrite 0x354ee5 thru 0x355e31 (0xf4d bytes) 000155[0] 1758734032.237962 mapread 0x68b146 thru 0x68ca96 (0x1951 bytes) 000156[0] 1758734032.245814 write 0x155d78 thru 0x155e2d (0xb6 bytes) 000157[1] 1758734032.251362 mapread 0x2cc002 thru 0x2cd233 (0x1232 bytes) 000161[0] 1758734032.288010 trunc from 0x9e15c8 to 0x68fa3c (0x351b8b bytes) 000162[1] 1758734032.319772 mapwrite 0x244cd4 thru 0x2451a2 (0x4cf bytes) 000163[0] 1758734032.422998 mapwrite 0x3d5bc2 thru 0x3d7888 (0x1cc7 bytes) 000164[1] 1758734032.512292 read 0x04bad5 thru 0x04d69d (0x1bc9 bytes) 000168[0] 1758734032.588538 mapwrite 0x445840 thru 0x44683d (0xffe bytes) 000169[1] 1758734032.636952 read 0x51c47c thru 0x51cd69 (0x8ee bytes) 000170[0] 1758734032.684025 read 0x363e31 thru 0x36527f (0x144f bytes) 000172[1] 1758734032.697707 read 0x3f09c8 thru 0x3f0ba2 (0x1db bytes) 000173[0] 1758734032.703488 read 0x32e45f thru 0x32f33c (0xede bytes) 000174[0] 1758734032.710558 mapread 0x6590cb thru 0x659a4c (0x982 bytes) 000176[1] 1758734032.718327 trunc from 0x68fa3d to 0x3d5b81 (0x2b9ebb bytes) 000177[0] 1758734032.772475 write 0x372dea thru 0x37489a (0x1ab1 bytes) 000179[1] 1758734032.818661 write 0x4e7b11 thru 0x4e8f4f (0x143f bytes) 000180[0] 1758734032.842106 trunc from 0x4e8f50 to 0x7976e9 (0x2ae79a bytes) 000182[1] 1758734032.930339 write 0x713aad thru 0x7140ab (0x5ff bytes) 000183[1] 1758734032.964529 write 0x430abf thru 0x4318be (0xe00 bytes) 000185[0] 1758734033.018743 write 0x3b3d0a thru 0x3b4521 (0x818 bytes) 000186[0] 1758734033.053146 mapread 0x4a571d thru 0x4a6acd (0x13b1 bytes) 000188[1] 1758734033.113512 write 0x63c93d thru 0x63d5bb (0xc7f bytes) 000189[0] 1758734033.161602 write 0x5d8ebb thru 0x5da9fc (0x1b42 bytes) 000190[1] 1758734033.214045 write 0x813692 thru 0x815358 (0x1cc7 bytes) truncating to largest ever: 0x987c3f 000197[0] 1758734033.361226 trunc from 0x815359 to 0x987c3e (0x1728e6 bytes) 000198[0] 1758734033.434269 write 0x6527c4 thru 0x654551 (0x1d8e bytes) 000199[0] 1758734033.467038 trunc from 0x987c3f to 0x7e3f4b (0x1a3cf3 bytes) 000204[0] 1758734033.521359 trunc from 0x7e3f4c to 0x565561 (0x27e9ea bytes) 000207[0] 1758734033.590431 read 0x2a9142 thru 0x2aa7dc (0x169b bytes) 000208[1] 1758734033.612460 trunc from 0x565562 to 0x8bb4dc (0x355f7b bytes) 000209[1] 1758734033.643032 write 0x1661e3 thru 0x16696c (0x78a bytes) 000214[0] 1758734033.715790 mapread 0x6dafb9 thru 0x6dbaeb (0xb33 bytes) 000215[1] 1758734033.747404 write 0x1c35d5 thru 0x1c38e2 (0x30e bytes) 000216[1] 1758734033.773053 read 0x3cebb2 thru 0x3cf642 (0xa91 bytes) 000218[0] 1758734033.788906 mapread 0x09f1ee thru 0x09fd93 (0xba6 bytes) 000219[1] 1758734033.816742 trunc from 0x8bb4dd to 0x1c4136 (0x6f73a6 bytes) 000221[1] 1758734033.893424 mapwrite 0x8b6d66 thru 0x8b6e27 (0xc2 bytes) 000224[1] 1758734033.993655 read 0x6392e7 thru 0x63ae13 (0x1b2d bytes) 000225[1] 1758734033.998231 mapread 0x02c182 thru 0x02c287 (0x106 bytes) 000228[1] 1758734034.019454 write 0x2d4067 thru 0x2d5d4b (0x1ce5 bytes) 000229[1] 1758734034.043361 write 0x837c8a thru 0x838db0 (0x1127 bytes) 000231[1] 1758734034.064515 write 0x41e1ad thru 0x41ea43 (0x897 bytes) 000233[0] 1758734034.092218 read 0x7fbe77 thru 0x7fd97a (0x1b04 bytes) 000234[1] 1758734034.141971 mapread 0x4340dd thru 0x4345c7 (0x4eb bytes) 000235[1] 1758734034.169485 mapwrite 0x69375a thru 0x6952fa (0x1ba1 bytes) 000236[0] 1758734034.277314 mapwrite 0x9d34b5 thru 0x9d461c (0x1168 bytes) 000237[1] 1758734034.389594 mapread 0x5993c1 thru 0x59b200 (0x1e40 bytes) 000239[0] 1758734034.426719 mapwrite 0x8de9ae thru 0x8e02eb (0x193e bytes) 000240[0] 1758734034.491303 mapwrite 0x4b5b27 thru 0x4b74cd (0x19a7 bytes) 000241[1] 1758734034.581161 write 0x0092d0 thru 0x0096af (0x3e0 bytes) 000242[0] 1758734034.614506 mapwrite 0x125434 thru 0x125f4d (0xb1a bytes) 000244[0] 1758734034.707430 mapread 0x8ead57 thru 0x8eb199 (0x443 bytes) 000245[0] 1758734034.716452 read 0x8afd32 thru 0x8b044c (0x71b bytes) 000251[1] 1758734034.770820 read 0x7d6898 thru 0x7d77ae (0xf17 bytes) 000252[0] 1758734034.801124 mapread 0x803a27 thru 0x804eda (0x14b4 bytes) 000255[0] 1758734034.833533 write 0x009366 thru 0x00ac9f (0x193a bytes) 000256[0] 1758734034.841119 trunc from 0x9d461d to 0x570950 (0x463ccc bytes) 000257[1] 1758734034.877109 mapread 0x496522 thru 0x496f92 (0xa71 bytes) 000262[0] 1758734034.949576 read 0x338c2e thru 0x339d1e (0x10f1 bytes) 000265[0] 1758734034.970593 read 0x205774 thru 0x206192 (0xa1f bytes) 000266[0] 1758734034.995336 mapwrite 0x8b9e8a thru 0x8baa76 (0xbed bytes) 000268[0] 1758734035.152543 read 0x201d9f thru 0x202b71 (0xdd3 bytes) 000271[0] 1758734035.196212 trunc from 0x8baa77 to 0x199ad6 (0x720fa0 bytes) 000276[0] 1758734035.243039 mapwrite 0x46b2a7 thru 0x46be92 (0xbec bytes) 000277[0] 1758734035.306351 read 0x17644e thru 0x1777f7 (0x13aa bytes) 000280[0] 1758734035.351918 mapread 0x3d59aa thru 0x3d7945 (0x1f9c bytes) 000282[1] 1758734035.365293 mapread 0x2daa00 thru 0x2daad4 (0xd5 bytes) 000284[1] 1758734035.409429 mapread 0x0a73d2 thru 0x0a7623 (0x252 bytes) 000287[0] 1758734035.463866 read 0x28af9d thru 0x28cc3b (0x1c9f bytes) 000288[0] 1758734035.485030 trunc from 0x46be93 to 0x6856bc (0x21982a bytes) 000289[0] 1758734035.532936 mapwrite 0x97c4a7 thru 0x97cb59 (0x6b3 bytes) 000291[1] 1758734035.608767 write 0x8cf4fb thru 0x8cf5fb (0x101 bytes) 000292[1] 1758734035.642494 trunc from 0x97cb5a to 0x76164e (0x21b50b bytes) truncating to largest ever: 0x9fca18 000294[0] 1758734035.700321 trunc from 0x76164f to 0x9fca17 (0x29b3c9 bytes) 000295[1] 1758734035.745190 write 0x3932af thru 0x393470 (0x1c2 bytes) 000296[1] 1758734035.776436 trunc from 0x9fca18 to 0x849a16 (0x1b3001 bytes) 000297[0] 1758734035.819995 mapwrite 0x2c8e2e thru 0x2ca352 (0x1525 bytes) 000298[1] 1758734035.919263 mapread 0x60e4bc thru 0x60f485 (0xfca bytes) 000299[0] 1758734035.965572 trunc from 0x849a17 to 0x23c7eb (0x60d22b bytes) 000300[0] 1758734036.010053 mapread 0x147e94 thru 0x14878c (0x8f9 bytes) 000301[1] 1758734036.019557 mapwrite 0x2765e3 thru 0x277b77 (0x1595 bytes) 000306[1] 1758734036.146129 mapwrite 0x6ce500 thru 0x6d02fb (0x1dfc bytes) 000307[1] 1758734036.257525 mapread 0x3eba86 thru 0x3ed639 (0x1bb4 bytes) 000309[1] 1758734036.310250 read 0x52b897 thru 0x52bdd2 (0x53c bytes) 000310[0] 1758734036.328810 mapwrite 0x572ccb thru 0x573932 (0xc68 bytes) 000311[0] 1758734036.415551 write 0x071c56 thru 0x07210b (0x4b6 bytes) 000313[0] 1758734036.452640 trunc from 0x6d02fc to 0x5d6933 (0xf99c8 bytes) 000314[1] 1758734036.490210 trunc from 0x5d6934 to 0x40d18f (0x1c97a4 bytes) 000315[0] 1758734036.533952 trunc from 0x40d190 to 0x060097 (0x3ad0f8 bytes) 000317[0] 1758734036.576601 write 0x80b318 thru 0x80cea6 (0x1b8f bytes) 000318[1] 1758734036.579787 trunc from 0x80cea7 to 0x4137b9 (0x3f96ed bytes) 000321[1] 1758734036.664353 read 0x2c00af thru 0x2c0504 (0x456 bytes) 000324[1] 1758734036.707106 mapwrite 0x9341db thru 0x9343d1 (0x1f7 bytes) 000325[0] 1758734036.791011 mapwrite 0x9788d4 thru 0x9789ea (0x117 bytes) 000329[0] 1758734036.873086 read 0x6cc625 thru 0x6cc6e7 (0xc3 bytes) 000330[0] 1758734036.878630 mapread 0x87c772 thru 0x87dafc (0x138b bytes) 000331[0] 1758734036.904190 read 0x6391b1 thru 0x63abf3 (0x1a43 bytes) 000332[1] 1758734036.907861 mapwrite 0x3e50d6 thru 0x3e5c88 (0xbb3 bytes) 000333[0] 1758734036.992494 mapwrite 0x4d9150 thru 0x4dac85 (0x1b36 bytes) 000334[1] 1758734037.075492 read 0x428ca7 thru 0x429741 (0xa9b bytes) 000345[0] 1758734037.222197 read 0x70c26a thru 0x70e0c1 (0x1e58 bytes) 000347[0] 1758734037.256772 trunc from 0x9789eb to 0x6246c0 (0x35432a bytes) 000349[1] 1758734037.319666 mapwrite 0x427faa thru 0x428905 (0x95c bytes) 000350[1] 1758734037.395995 read 0x33f3c6 thru 0x340366 (0xfa1 bytes) 000353[0] 1758734037.438144 write 0x554b61 thru 0x556798 (0x1c38 bytes) 000355[1] 1758734037.454135 mapread 0x2a107a thru 0x2a1be2 (0xb69 bytes) 000359[1] 1758734037.490277 trunc from 0x6246c1 to 0x32db6c (0x2f6b54 bytes) 000361[1] 1758734037.538495 write 0x835b03 thru 0x8360b0 (0x5ae bytes) 000362[0] 1758734037.555957 write 0x6fa297 thru 0x6fb003 (0xd6d bytes) 000366[1] 1758734037.625985 write 0x09685d thru 0x096fa9 (0x74d bytes) 000368[0] 1758734037.656571 write 0x84dbea thru 0x84dcf6 (0x10d bytes) 000373[1] 1758734037.696187 read 0x0bd5d6 thru 0x0be760 (0x118b bytes) 000378[0] 1758734037.741609 mapwrite 0x62de35 thru 0x62e049 (0x215 bytes) 000381[0] 1758734037.826110 write 0x95ac43 thru 0x95bfc2 (0x1380 bytes) 000382[1] 1758734037.861684 mapwrite 0x85864c thru 0x85a26e (0x1c23 bytes) 000384[0] 1758734037.963105 write 0x26890e thru 0x26a200 (0x18f3 bytes) 000385[0] 1758734037.995436 write 0x0a9d1b thru 0x0aa07a (0x360 bytes) 000386[1] 1758734038.018934 read 0x77be03 thru 0x77c4e0 (0x6de bytes) 000387[0] 1758734038.048682 trunc from 0x95bfc3 to 0x90afe5 (0x50fdd bytes) 000388[1] 1758734038.092665 mapwrite 0x7cc470 thru 0x7ccd32 (0x8c3 bytes) 000389[1] 1758734038.167829 read 0x25c9ab thru 0x25dcdd (0x1333 bytes) 000390[1] 1758734038.183191 mapwrite 0x09ca2d thru 0x09dda3 (0x1377 bytes) 000391[0] 1758734038.251263 write 0x79e309 thru 0x79ef5a (0xc52 bytes) 000392[1] 1758734038.278338 mapwrite 0x3720b6 thru 0x3736de (0x1629 bytes) 000395[1] 1758734038.369609 mapwrite 0x560f87 thru 0x561e7c (0xef6 bytes) 000396[1] 1758734038.421579 mapwrite 0x34a313 thru 0x34b82d (0x151b bytes) 000397[0] 1758734038.473203 mapread 0x07474e thru 0x074f93 (0x846 bytes) 000398[0] 1758734038.498345 read 0x584f6a thru 0x586c03 (0x1c9a bytes) 000402[1] 1758734038.512525 mapread 0x2bd163 thru 0x2bdb87 (0xa25 bytes) 000410[1] 1758734038.521054 trunc from 0x90afe6 to 0x3cd55b (0x53da8a bytes) 000412[1] 1758734038.599952 trunc from 0x3cd55c to 0x1dc38c (0x1f11cf bytes) 000415[0] 1758734038.638665 mapread 0x05e646 thru 0x0602d9 (0x1c94 bytes) 000416[0] 1758734038.691939 mapread 0x1a84fb thru 0x1a979d (0x12a3 bytes) 000417[0] 1758734038.720922 mapwrite 0x266560 thru 0x266ada (0x57b bytes) 000418[0] 1758734038.821704 trunc from 0x266adb to 0x4deb67 (0x27808d bytes) 000419[1] 1758734038.839083 write 0x8ea2e4 thru 0x8ec1b0 (0x1ecd bytes) 000422[1] 1758734038.899383 mapread 0x2cf1b6 thru 0x2d1139 (0x1f84 bytes) 000423[1] 1758734038.936481 write 0x2dd0bc thru 0x2dd34e (0x293 bytes) 000426[1] 1758734038.982070 write 0x60986b thru 0x60a15b (0x8f1 bytes) 000428[1] 1758734039.014073 mapwrite 0x920dc4 thru 0x921381 (0x5be bytes) 000429[1] 1758734039.092754 write 0x28372c thru 0x284419 (0xcee bytes) 000433[0] 1758734039.148447 trunc from 0x921382 to 0x00c01e (0x915363 bytes) 000436[0] 1758734039.227666 mapwrite 0x529a89 thru 0x52a2ef (0x867 bytes) 000437[0] 1758734039.308993 trunc from 0x52a2f0 to 0x457aa7 (0xd2848 bytes) 000438[1] 1758734039.338214 mapread 0x17294e thru 0x173ffc (0x16af bytes) 000441[1] 1758734039.408288 read 0x353719 thru 0x354b93 (0x147b bytes) 000442[1] 1758734039.430766 trunc from 0x457aa8 to 0x96c8ab (0x514e04 bytes) 000444[0] 1758734039.488939 mapwrite 0x23278a thru 0x23366f (0xee6 bytes) 000446[0] 1758734039.597680 mapread 0x43669d thru 0x4377f7 (0x115b bytes) 000447[0] 1758734039.628625 trunc from 0x96c8ac to 0x86f9cf (0xfcedc bytes) 000448[1] 1758734039.663671 trunc from 0x86f9d0 to 0x41f1d9 (0x4507f6 bytes) 000449[0] 1758734039.700268 read 0x21b636 thru 0x21cba5 (0x1570 bytes) 000450[0] 1758734039.721807 read 0x0a10d2 thru 0x0a1659 (0x588 bytes) 000456[0] 1758734039.769308 mapwrite 0x344d4e thru 0x3452ca (0x57d bytes) 000461[0] 1758734039.929132 write 0x9573be thru 0x958a4f (0x1692 bytes) 000463[1] 1758734039.962853 mapwrite 0x7d115b thru 0x7d1c76 (0xb1c bytes) 000464[0] 1758734040.040916 trunc from 0x958a50 to 0x7aef25 (0x1a9b2a bytes) 000465[1] 1758734040.076482 mapread 0x20c92a thru 0x20ced3 (0x5aa bytes) 000466[0] 1758734040.096694 trunc from 0x7aef26 to 0x0fbbc9 (0x6b335c bytes) 000470[1] 1758734040.155445 read 0x0c4f1f thru 0x0c5155 (0x237 bytes) 000472[1] 1758734040.178159 read 0x086881 thru 0x087d26 (0x14a6 bytes) 000473[0] 1758734040.178633 read 0x06add0 thru 0x06cd71 (0x1fa2 bytes) 000476[0] 1758734040.196530 mapread 0x0f4fd8 thru 0x0f5e44 (0xe6d bytes) 000478[1] 1758734040.197559 trunc from 0x0fbbca to 0x3c7f4f (0x2cc386 bytes) 000479[1] 1758734040.231212 mapread 0x0ce3fd thru 0x0cfb9e (0x17a2 bytes) 000481[1] 1758734040.290559 mapread 0x1e4c39 thru 0x1e61d0 (0x1598 bytes) 000496[0] 1758734040.366275 mapwrite 0x25ca35 thru 0x25e414 (0x19e0 bytes) 000498[1] 1758734040.499016 write 0x31f124 thru 0x31fc3c (0xb19 bytes) 000499[0] 1758734040.532172 trunc from 0x3c7f50 to 0x6445ee (0x27c69f bytes) All operations completed A-OK! PASS 16b (18s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 13:14:03 (1758734043) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 13:14:05 (1758734045) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.280632 s, 29.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.24365 s, 34.4 MB/s ldlm.namespaces.MGC192.168.203.130@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d17881f1000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d179dd36800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d179dd36800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d179dd36800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.18406 s, 17.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.51337 s, 69.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.16493 s, 90.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.78921 s, 37.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.52549 s, 23.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.55429 s, 67.5 MB/s PASS 16d (24s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 13:14:29 (1758734069) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.1127 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484605 s, 21.6 MB/s PASS 16e (4s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 13:14:33 (1758734073) TIMEOUT PASS 16f (23s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 13:14:57 (1758734097) TIMEOUT PASS 16g (24s) == sanityn test 16h: mmap read after truncate file ======= 13:15:21 (1758734121) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 13:15:24 (1758734124) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (4s) == sanityn test 16j: race dio with buffered i/o ========== 13:15:28 (1758734128) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.645306 s, 40.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.590555 s, 44.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.35449 s, 19.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.35777 s, 73.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.26969 s, 20.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.352865 s, 74.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29987 s, 20.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.803337 s, 32.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.919098 s, 28.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.48357 s, 17.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57093 s, 16.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.482417 s, 54.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.647386 s, 40.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55145 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67443 s, 15.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.595472 s, 44.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.741144 s, 35.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5422 s, 17.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.69233 s, 15.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (19s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 13:15:47 (1758734147) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed Sep 24 13:16:05 EDT 2025 + fsx 2 done: rc=0 Wed Sep 24 13:16:05 EDT 2025 All operations completed A-OK! + fsx 3 done: rc=0 Wed Sep 24 13:16:06 EDT 2025 + fsx 4 done: rc=0 Wed Sep 24 13:16:06 EDT 2025 + fsx 5 done: rc=0 Wed Sep 24 13:16:06 EDT 2025 PASS 16k (25s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 13:16:13 (1758734173) fail_loc=0x8000030a PASS 17 (7s) == sanityn test 18: mmap sanity check =========================================================================================== 13:16:20 (1758734180) mmap test1: basic mmap operation (PASS, 0.12193s) mmap test2: MAP_PRIVATE not write back (PASS, 0.058635s) mmap test3: concurrent mmap ops on two nodes (PASS, 21.225s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3231s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1314s) mmap test6: check mmap write/read content on two nodes (PASS, 0.15541s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.70903s) mmap test11: multi-thread shared mmap (PASS, 0.82357s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0646s) PASS 18 (33s) == sanityn test 19: test concurrent uncached read races ========================================================================= 13:16:54 (1758734214) oleg330-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 13:16:56 (1758734216) PASS 20 (4s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 13:17:00 (1758734220) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 13:17:04 (1758734224) now is 1758734225 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 new atime is 1758734286 PASS 23 (66s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 13:18:09 (1758734289) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 13:18:12 (1758734292) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 13:18:15 (1758734295) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (4s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 13:18:19 (1758734299) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 13:18:21 (1758734301) utime: good mknod times 1758734301 <= 1758734302 <= 1758734302 for /mnt/lustre/f26a utime: good mknod times 1758734301 <= 1758734302 <= 1758734302 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 13:18:25 (1758734305) PASS 26b (6s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 13:18:31 (1758734311) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-09-24 13:18:31.000000000 -0400 Birth: 2025-09-24 13:18:31.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-09-24 13:18:31.000000000 -0400 Birth: 2025-09-24 13:18:31.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 13:18:36 (1758734316) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 13:18:41 (1758734321) fail_loc=0x314 PASS 31a (5s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 13:18:46 (1758734326) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg330-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.130@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 13:19:07 (1758734347) -rw-r--r-- 1 root root 0 Sep 24 13:19 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 13:19:13 (1758734353) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42971 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 13:19:17 (1758734357) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 13:19:22 (1758734362) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 13:19:24 (1758734364) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 13:19:26 (1758734366) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 13:19:27 (1758734367) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 13:19:28 (1758734368) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid in FULL state after 0 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid in IDLE state after 0 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid in FULL state after 0 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid in IDLE state after 0 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d17881f1000.ost_server_uuid in IDLE state after 0 sec oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d17881f1000.ost_server_uuid in IDLE state after 0 sec PASS 34 (98s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 13:21:06 (1758734466) fail_loc=0x80000317 Race attempt 0 Wait for 49097 49230 for 60 sec... - open/close 2005 (time 1758734479.36 total 10.00 last 200.47) - open/close 3831 (time 1758734489.36 total 20.00 last 182.58) total: 4000 open/close in 20.91 seconds: 191.29 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.130@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.130@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 13:22:12 (1758734532) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg330-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.512019 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.514435 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.477828 s, 110 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.452442 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.514404 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.492131 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.476005 s, 110 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.487246 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.484423 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.488371 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.494938 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7444480) after(7495680) PASS 36 (233s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 13:26:05 (1758734765) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 - create 3773 (time 1758734776.92 total 10.00 last 377.24) - create 7379 (time 1758734786.93 total 20.00 last 360.49) total: 10000 create in 26.39 seconds: 378.96 ops/second PASS 37 (31s) == sanityn test 39a: file mtime does not change after rename ========================================================== 13:26:36 (1758734796) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 13:26:39 (1758734799) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 13:26:42 (1758734802) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 13:26:46 (1758734806) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 13:26:50 (1758734810) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 13:26:59 (1758734819) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 13:27:09 (1758734829) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 13:27:18 (1758734838) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 13:27:26 (1758734846) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 13:27:32 (1758734852) open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 13:27:38 (1758734858) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 13:27:44 (1758734864) Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 13:27:50 (1758734870) Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 13:27:56 (1758734876) Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 13:28:02 (1758734882) Conflict PASS 41g (6s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 13:28:08 (1758734888) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 13:28:15 (1758734895) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (746s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 13:40:40 (1758735640) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 13:40:46 (1758735646) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 13:40:52 (1758735652) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 13:40:58 (1758735658) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 13:41:03 (1758735663) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 13:41:10 (1758735670) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 13:41:16 (1758735676) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 13:41:22 (1758735682) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 13:41:29 (1758735689) PASS 43a (25s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 13:41:54 (1758735714) Conflict PASS 43b (7s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 13:42:01 (1758735721) Conflict PASS 43c (6s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 13:42:07 (1758735727) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 13:42:13 (1758735733) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 13:42:18 (1758735738) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (6s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 13:42:24 (1758735744) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 13:42:30 (1758735750) Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 13:42:37 (1758735757) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 13:42:38 (1758735758) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (49s) == sanityn test 43k: unlink vs create ==================== 13:43:27 (1758735807) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (479s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 13:51:26 (1758736286) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 24 13:51:29 EDT 2025 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:51:31 (1758736291) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:51:37 (1758736297) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:51:42 (1758736302) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:51:48 (1758736308) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:51:54 (1758736314) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:51:59 (1758736319) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:52:05 (1758736325) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:52:10 (1758736330) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:52:11 (1758736331) PASS 45a (32s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:52:43 (1758736363) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 13:52:49 (1758736369) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 13:52:54 (1758736374) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 13:52:59 (1758736379) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 13:53:04 (1758736384) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 13:53:10 (1758736390) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 13:53:15 (1758736395) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 13:53:19 (1758736399) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 13:53:20 (1758736400) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (628s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:03:48 (1758737028) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:03:59 (1758737039) open: File exists Conflict PASS 46b (9s) == sanityn test 46c: pdirops: link vs link ======================================================================== 14:04:08 (1758737048) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 14:04:16 (1758737056) Conflict PASS 46d (8s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 14:04:24 (1758737064) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 14:04:32 (1758737072) Conflict PASS 46f (8s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 14:04:40 (1758737080) Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 14:04:48 (1758737088) Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 14:04:56 (1758737096) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 14:04:59 (1758737099) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 14:05:01 (1758737101) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 14:05:03 (1758737103) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 14:05:05 (1758737105) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 14:05:06 (1758737106) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 14:05:08 (1758737108) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 14:05:10 (1758737110) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 14:05:12 (1758737112) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0628377 s, 163 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (9s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 14:05:21 (1758737121) 0+1 records in 0+1 records out 158 bytes copied, 0.0211978 s, 7.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260083) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 14:05:26 (1758737126) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0078127 s, 131 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499408 s, 21.0 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 14:05:42 (1758737142) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0170879 s, 59.9 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 14:05:50 (1758737150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333223 s, 31.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 14:05:55 (1758737155) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 14:06:00 (1758737160) llite.lustre-ffff8d17881f1000.xattr_cache=1 llite.lustre-ffff8d179dd36800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:01.000000000 -0400 Modify: 2025-09-24 14:06:01.000000000 -0400 Change: 2025-09-24 14:06:01.000000000 -0400 Birth: 2025-09-24 14:06:01.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:07.000000000 -0400 Modify: 2025-09-24 14:06:07.000000000 -0400 Change: 2025-09-24 14:06:07.000000000 -0400 Birth: 2025-09-24 14:06:07.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:13.000000000 -0400 Modify: 2025-09-24 14:06:13.000000000 -0400 Change: 2025-09-24 14:06:13.000000000 -0400 Birth: 2025-09-24 14:06:13.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:13.000000000 -0400 Modify: 2025-09-24 14:06:13.000000000 -0400 Change: 2025-09-24 14:06:13.000000000 -0400 Birth: 2025-09-24 14:06:13.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:18.000000000 -0400 Modify: 2025-09-24 14:06:19.000000000 -0400 Change: 2025-09-24 14:06:19.000000000 -0400 Birth: 2025-09-24 14:06:18.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:06:18.000000000 -0400 Modify: 2025-09-24 14:06:19.000000000 -0400 Change: 2025-09-24 14:06:19.000000000 -0400 Birth: 2025-09-24 14:06:18.000000000 -0400 PASS 54 (27s) == sanityn test 55a: rename vs unlink target dir ========= 14:06:27 (1758737187) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 14:06:38 (1758737198) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 14:06:47 (1758737207) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 14:07:03 (1758737223) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 14:07:14 (1758737234) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 14:07:16 (1758737236) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (15s) == sanityn test 55g: rename: race with trylock =========== 14:07:31 (1758737251) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (18s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 14:07:49 (1758737269) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1758737270 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.73541 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (90s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 14:09:19 (1758737359) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1758737364 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.79521 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (79s) == sanityn test 60: Verify data_version behaviour ======== 14:10:38 (1758737438) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 14:10:44 (1758737444) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 14:10:50 (1758737450) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 14:10:57 (1758737457) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0114105 s, 5.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 14:10:59 (1758737459) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0209869 s, 3.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 14:11:02 (1758737462) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 14:11:04 (1758737464) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0148967 s, 4.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 14:11:06 (1758737466) llite.lustre-ffff8d17881f1000.xattr_cache=1 llite.lustre-ffff8d179dd36800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 14:11:11 (1758737471) llite.lustre-ffff8d17881f1000.xattr_cache=1 llite.lustre-ffff8d179dd36800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 14:11:16 (1758737476) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0288531 s, 35.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0219579 s, 46.6 kB/s 272874: get lock2 272875: get lock1 272874: done 272874: get lock1 272875: done 272875: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 272875: cannot get lock2: Resource deadlock avoided 272875: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 272874: cannot get lock1: Resource deadlock avoided 272874: done 272874: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 14:11:25 (1758737485) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.034575 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269882 s, 38.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273456 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (10s) == sanityn test 76: Verify MDT open_files listing ======== 14:11:36 (1758737496) opening files: ........ closing files: ........ PASS 76 (49s) == sanityn test 77a: check FIFO NRS policy =============== 14:12:26 (1758737546) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.364735 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543614 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056445 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316714 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433966 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100844 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176526 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176626 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108328 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19323 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171437 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198486 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253221 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228516 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240337 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239253 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308506 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459983 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767076 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106082 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119388 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570077 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00191415 s, 548 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133161 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106372 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705512 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354912 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137385 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447041 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868559 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412071 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298488 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322429 s, 32.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 14:12:34 (1758737554) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.451232 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555411 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670477 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170051 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669923 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746961 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198228 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155451 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218251 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210009 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254019 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217445 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262377 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319005 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311936 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342979 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389839 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104904 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00233143 s, 450 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584805 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146011 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826449 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154007 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148692 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841144 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148396 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142585 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503977 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01303 s, 80.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392187 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164158 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321494 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471811 s, 22.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.342861 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362342 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725367 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710103 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12053 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19234 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240215 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25066 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256205 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322851 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365449 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285161 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366754 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388098 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297269 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391358 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416818 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864315 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555928 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00519584 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592118 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591749 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604178 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614783 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219046 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215182 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00559438 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606933 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688714 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775763 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230433 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113409 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784011 s, 13.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (11s) == sanityn test 77c: check ORR NRS policy ================ 14:12:45 (1758737565) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.362815 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383711 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664377 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822479 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203974 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215898 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218265 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210356 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217936 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19669 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218383 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261572 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240917 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214418 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271704 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233123 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284492 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529005 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124625 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03577 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451876 s, 232 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00503013 s, 208 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495951 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060153 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344438 s, 30.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467882 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192395 s, 54.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279171 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044524 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185805 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214012 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175981 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018164 s, 57.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.285316 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359937 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683106 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14331 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100744 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16146 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116389 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168228 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185686 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205475 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166344 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159301 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193417 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173777 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214651 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2032 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222734 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250536 s, 41.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033795 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517879 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079961 s, 131 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250441 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253242 s, 414 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134024 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083454 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995008 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713455 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114131 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480609 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413313 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375224 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344652 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186363 s, 56.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315821 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651081 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510899 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432267 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562077 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116739 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149401 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144831 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123811 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929516 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19391 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246981 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241803 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276601 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637914 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26875 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121038 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294126 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824465 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106042 s, 98.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287573 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010912 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143272 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436061 s, 240 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00885873 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536339 s, 196 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00596937 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572956 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156873 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767308 s, 137 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157713 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00659459 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455652 s, 23.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (12s) == sanityn test 77d: check TRR nrs policy ================ 14:12:57 (1758737577) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248397 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297161 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712772 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672479 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811018 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150879 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136736 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181867 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19385 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213331 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209805 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198122 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222572 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181084 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200045 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187097 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17437 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240109 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132031 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367975 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240295 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576833 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582009 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507481 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267871 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387241 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214077 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358724 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227557 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206804 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199019 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193517 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171729 s, 61.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.226061 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403091 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442272 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126112 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146133 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129966 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144616 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158317 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15373 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154718 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1585 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151389 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166622 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153956 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185168 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154272 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18199 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289217 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178838 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385656 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344769 s, 304 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610037 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434539 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712944 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476685 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303764 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271631 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152401 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238923 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224417 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115397 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824918 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195573 s, 53.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20966 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418723 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494529 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114006 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161266 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148769 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143673 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183038 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188212 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151567 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188704 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186754 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169755 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206751 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196053 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206833 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219442 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280066 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721621 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00229422 s, 457 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106152 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101705 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788346 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550378 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16746 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541354 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444167 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522953 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417162 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035524 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010218 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314187 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177017 s, 59.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (9s) == sanityn test 77e: check TBF NID nrs policy ============ 14:13:06 (1758737586) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.30@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254589 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433332 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856205 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601549 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111789 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119306 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972545 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943224 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122508 s, 8.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962949 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133037 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139547 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123221 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113629 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117332 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104204 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152899 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152942 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175258 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182291 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164214 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162178 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153536 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161213 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179911 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183627 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166106 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16573 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173016 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20393 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173354 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17249 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180114 s, 5.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.213312 s, 78.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426711 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037532 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318198 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509709 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114351 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140273 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115148 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139489 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171243 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165471 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162458 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170092 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169679 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18234 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143611 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135686 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129824 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117779 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134686 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126903 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119449 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120263 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124034 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135406 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135927 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130825 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135825 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150454 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139957 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137613 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14539 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150767 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.316088 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307577 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440724 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873546 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108609 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101412 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125504 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131663 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122288 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141322 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145827 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122557 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125614 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151627 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138645 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136405 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189679 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227818 s, 46.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371431 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335185 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456364 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499286 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346393 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369879 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424009 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429271 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352387 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281404 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291399 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030855 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313105 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343088 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617886 s, 17.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.238017 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758525 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323459 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803623 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674919 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112037 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867248 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982855 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116292 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999101 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131324 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15908 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171801 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214895 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216415 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214273 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217825 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00559813 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612011 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805264 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552415 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631647 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454539 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482005 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437202 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601263 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318293 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459136 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406945 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296288 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024413 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127439 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158848 s, 66.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (14s) == sanityn test 77f: check TBF JobID nrs policy ========== 14:13:20 (1758737600) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.221668 s, 75.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428898 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518301 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710471 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144286 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153043 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152491 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131417 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150351 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130344 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163654 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153077 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16034 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176603 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17016 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172241 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296842 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340688 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128175 s, 81.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394502 s, 26.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152934 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169974 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140789 s, 74.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238997 s, 43.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276666 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294659 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302507 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303565 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336276 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343039 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348103 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358769 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459025 s, 22.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.210309 s, 79.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532518 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324484 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632189 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114624 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147618 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140215 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141048 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167479 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148253 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163629 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180117 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185548 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196013 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205245 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196615 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231295 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369408 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00590131 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279002 s, 376 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699277 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524296 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709586 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061339 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378359 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518404 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727773 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459446 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165946 s, 63.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115728 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311891 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200668 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266783 s, 39.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335879 s, 50.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358323 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528557 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101103 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951721 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169336 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185887 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14276 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186544 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186846 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152368 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145138 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137841 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195161 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201312 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196423 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169814 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d17881f1000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199149 s, 52.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350156 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471161 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238066 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477321 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365671 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552828 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303585 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031011 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494745 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285562 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203216 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209145 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112623 s, 93.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199897 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214146 s, 49.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234663 s, 71.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660688 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649972 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577375 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799096 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626212 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893472 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623527 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656752 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936579 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080371 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133219 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439863 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378408 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425921 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156639 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623164 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460195 s, 228 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746754 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151509 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470571 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340723 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334092 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547706 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379476 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396514 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045953 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420026 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480855 s, 218 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8d17881f1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218576 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200897 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144282 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230986 s, 4.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (15s) == sanityn test 77g: Change TBF type directly ============ 14:13:35 (1758737615) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (7s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 14:13:42 (1758737622) oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg330-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg330-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg330-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg330-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 PASS 77h (7s) == sanityn test 77i: Change rank of TBF rule ============= 14:13:49 (1758737629) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (15s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 14:14:04 (1758737644) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.339409 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339414 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451707 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112707 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122449 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112515 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11817 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13689 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148024 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135688 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169655 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140882 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139926 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210619 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217284 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272837 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26153 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439241 s, 239 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536646 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452986 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030071 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433271 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337678 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320879 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207578 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377298 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698873 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539712 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00205665 s, 510 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511332 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529059 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407757 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667417 s, 157 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.1119 s, 17.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5829 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (60s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 14:15:05 (1758737705) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2669 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850591 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974964 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110801 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118024 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105272 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134203 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134543 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129741 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152429 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155829 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179292 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204698 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206984 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212983 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240602 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248998 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431387 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422658 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361187 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416002 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449704 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479064 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543432 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562633 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576962 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606599 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584596 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696971 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640803 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673483 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763799 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750789 s, 14.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9117 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9241 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25174 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029589 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025399 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578905 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102576 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724052 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112153 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800768 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916834 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148212 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1491 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172667 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18968 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155482 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205922 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211464 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182622 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426559 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456824 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461437 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438091 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507649 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507956 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531712 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534485 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539171 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540868 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572857 s, 18.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533142 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540315 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606206 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636629 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678438 s, 15.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7519 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9282 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (138s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 14:17:23 (1758737843) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26384 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767998 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980872 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087265 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12852 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167939 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207413 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199586 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221169 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219328 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217629 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211457 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193006 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219167 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171766 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230704 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260825 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404727 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455719 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505014 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366256 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416052 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496813 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377414 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051534 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448807 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577441 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601567 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602497 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688691 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530877 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567342 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632044 s, 16.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8449 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.979 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24902 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330826 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494143 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567603 s, 18.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582477 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106365 s, 9.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933134 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105172 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102574 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123731 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125851 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152401 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165069 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164379 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180809 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188321 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180106 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435509 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493179 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046941 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464371 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449038 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511284 s, 20.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503168 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540221 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605263 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576097 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589539 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588117 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620507 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671552 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670842 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704458 s, 14.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8147 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9905 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (141s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 14:19:45 (1758737985) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410827 s, 40.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359155 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332975 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543203 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555486 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144333 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142867 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131699 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162676 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194895 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193524 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152344 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175012 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195522 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178004 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163705 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17785 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514394 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148422 s, 70.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413701 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517956 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139805 s, 75.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447965 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351851 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557514 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266689 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216917 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061317 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681393 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671296 s, 156 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162218 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173585 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229113 s, 45.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.62299 s, 18.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83776 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.35141 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389968 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438623 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160969 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135945 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161485 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159196 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162251 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1352 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177531 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159593 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147541 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147164 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173231 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166099 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230029 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216056 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037251 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167597 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041116 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633979 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511651 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534343 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575834 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561921 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761324 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067306 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464208 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795536 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630102 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851987 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143583 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137434 s, 7.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.5983 s, 18.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8736 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.30435 s, 55.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381342 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390621 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046925 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575233 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086169 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113805 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124398 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106212 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122182 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129861 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140367 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124176 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131124 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151635 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121356 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153259 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575748 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303913 s, 345 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811362 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402291 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112894 s, 92.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483812 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0084639 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363049 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00610945 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484371 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511443 s, 205 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386274 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325391 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635863 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902441 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00742757 s, 141 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.5992 s, 18.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96325 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496083 s, 33.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334079 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218167 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394539 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752667 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124123 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141824 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131951 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136854 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141649 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139974 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146487 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132095 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137833 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145443 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141074 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161131 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166128 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564358 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043482 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054277 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072908 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063719 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579866 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879638 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432317 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279644 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247383 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169014 s, 62.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208863 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192925 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118114 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717079 s, 146 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9268 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81782 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.42237 s, 19.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82171 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.898956 s, 18.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435593 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874133 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663926 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706428 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660464 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118309 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998847 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114523 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959858 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153014 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186306 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179066 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199337 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169362 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201196 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181817 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045856 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318545 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194203 s, 54.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276088 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220932 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424762 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212432 s, 49.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165515 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340298 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137696 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391142 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171711 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304197 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00291862 s, 359 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339051 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568733 s, 184 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6448 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5182 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300316 s, 55.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027432 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314335 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117896 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122469 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119377 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126554 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130968 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121419 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129294 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132459 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11573 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13866 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123446 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134884 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128948 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155183 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181076 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414464 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178045 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380257 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186506 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004127 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612673 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551852 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069833 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514411 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677131 s, 155 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381273 s, 27.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057586 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508211 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440094 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255735 s, 41.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51447 s, 19.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84963 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518059 s, 32.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367535 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508566 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131644 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118227 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132847 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139837 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141342 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128829 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113201 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115398 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125311 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145746 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217645 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228176 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409913 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425941 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262295 s, 400 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632217 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356098 s, 294 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751072 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114837 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753559 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368257 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440071 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09655 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547493 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106553 s, 98.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402913 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871649 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694884 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124911 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396658 s, 26.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90605 s, 10.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81597 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88635 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85044 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (372s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 14:25:57 (1758738357) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392845 s, 42.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038749 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601243 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109305 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131027 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10731 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148324 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152791 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125411 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121936 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156078 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145749 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176399 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173857 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162292 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172409 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23312 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449286 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172229 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453689 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591939 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746992 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823244 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828582 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463998 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678154 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408291 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418761 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278707 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294054 s, 357 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230977 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251448 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168925 s, 62.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.63099 s, 18.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84334 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (41s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 14:26:39 (1758738399) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.305207 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364389 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041638 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100993 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102847 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11389 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101785 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129155 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142373 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13369 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141868 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154392 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161359 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18101 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191264 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241059 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249031 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274276 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303013 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244014 s, 43.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242482 s, 43.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319004 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251754 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484177 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281253 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359034 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286711 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374012 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307351 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346049 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341431 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391422 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380154 s, 27.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.30919 s, 19.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81187 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (40s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 14:27:18 (1758738438) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.297092 s, 56.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308932 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298808 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303976 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382063 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686765 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069194 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968078 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115608 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11566 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104217 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101139 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12105 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118865 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12902 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132969 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128162 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929552 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980503 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168706 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436317 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377502 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465027 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419792 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397589 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333041 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588407 s, 178 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231607 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225032 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021986 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104517 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214346 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172993 s, 60.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.48037 s, 19.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01553 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (37s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 14:27:55 (1758738475) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.541975 s, 31.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538948 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696123 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919722 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797715 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491091 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489525 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830964 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138157 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149734 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149623 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172913 s, 6.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161639 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145697 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187584 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142921 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152208 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226102 s, 464 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967679 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628119 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832812 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796903 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581672 s, 18.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104027 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742212 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376514 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472903 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384548 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466798 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619207 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376308 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334263 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273863 s, 38.3 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98711 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86956 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.80099 s, 18.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86343 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 14:29:16 (1758738556) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.906057 s, 18.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410305 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600812 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657363 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770844 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10641 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829988 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950572 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108586 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136469 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180206 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187031 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202845 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193034 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221112 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233657 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249943 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020369 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331072 s, 317 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441131 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354282 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436301 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364599 s, 28.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348475 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032562 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390789 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479894 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477333 s, 220 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279204 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475912 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316093 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049231 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457532 s, 22.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6393 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9466 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (69s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 14:30:25 (1758738625) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.515532 s, 32.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398951 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044877 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314959 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802201 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996257 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119602 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12151 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137316 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138766 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145252 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1172 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139738 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135169 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146472 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146525 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133769 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00849769 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142844 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395123 s, 265 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572058 s, 183 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00576803 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667668 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456395 s, 230 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426486 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374445 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478283 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400967 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559081 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238945 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186671 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665524 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784892 s, 134 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.48267 s, 19.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87033 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg330-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501653 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402201 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614178 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510051 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119123 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129768 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146522 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115153 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17768 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123784 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162111 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159317 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176929 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185185 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185015 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166644 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174955 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201644 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245512 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399384 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230076 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268968 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312235 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412237 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247484 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499415 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384479 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411415 s, 25.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349893 s, 30.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409395 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642173 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728618 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724586 s, 14.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0701 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89949 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97427 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89239 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (126s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 14:32:31 (1758738751) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (9s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 14:32:40 (1758738760) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg330-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.97903 s, 264 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg330-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0706 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (55s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 14:33:35 (1758738815) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.400027 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692352 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429438 s, 24.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856153 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04584 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043772 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10449 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103616 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948018 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207169 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196751 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224608 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226562 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254909 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227246 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246663 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921259 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989907 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116229 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414645 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106712 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993505 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09399 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987345 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10711 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102708 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106436 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11142 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999092 s, 10.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995461 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547407 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326584 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610398 s, 17.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg330-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg330-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.74819 s, 18.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08774 s, 20.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.277705 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318417 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487231 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993632 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100988 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945789 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849184 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102908 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155476 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971967 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170199 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184107 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129425 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223963 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173156 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171407 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200104 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887143 s, 118 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203914 s, 51.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204058 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159224 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361009 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230307 s, 45.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563887 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026148 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275687 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390434 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330937 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346745 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361085 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369974 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429104 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493426 s, 21.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg330-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg330-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.26258 s, 16.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0173 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (79s) == sanityn test 77o: Changing rank should not panic ====== 14:34:54 (1758738894) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 14:35:04 (1758738904) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (79s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 14:36:23 (1758738983) mds.MDS.mdt.nrs_policies=tbf oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg330-client: oleg330-server: ssh exited with exit code 36 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 oleg330-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg330-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg330-client: oleg330-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (23s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 14:36:46 (1758739006) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 307920 kill md_thread xargs PASS 77r (48s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 14:37:34 (1758739054) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (8s) == sanityn test 79: xattr: intent error ================== 14:37:42 (1758739062) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505967 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:37:44.000000000 -0400 Modify: 2025-09-24 14:37:44.000000000 -0400 Change: 2025-09-24 14:37:44.000000000 -0400 Birth: 2025-09-24 14:37:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (16s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 14:37:58 (1758739078) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 14:38:00 (1758739080) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 14:38:03 (1758739083) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 14:38:06 (1758739086) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 14:38:08 (1758739088) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 14:38:10 (1758739090) - open/close 832 (time 1758739105.12 total 10.01 last 83.13) - open/close 1704 (time 1758739115.12 total 20.01 last 87.20) total: 2500 open/close in 29.29 seconds: 85.36 ops/second snapshot_time 1758739190.821794530 secs.nsecs start_time 1758739093.116212506 secs.nsecs elapsed_time 97.705582024 secs.nsecs open 2505 samples [usecs] 132 15504 2221758 3259865294 close 2504 samples [usecs] 152 5906 821554 552350496 mknod 2501 samples [usecs] 429 12358 1926038 2437704196 unlink 2500 samples [usecs] 536 38806 3528781 18590272751 mkdir 2 samples [usecs] 882 1927 2809 4491253 rmdir 2 samples [usecs] 6189 11525 17714 171129346 rename 2500 samples [usecs] 346 49266 10908880 95939604090 getattr 3761 samples [usecs] 8 5233 164705 104247001 parallel_rename_file 2500 samples [usecs] 346 49266 10908880 95939604090 crossdir_rename 2500 samples [usecs] 346 49266 10908880 95939604090 PASS 81d (105s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 14:39:55 (1758739195) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7550 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7550 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 14:40:02 (1758739202) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 14:40:05 (1758739205) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 14:40:19 (1758739219) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1758739221 Finishing test test1 at 1758739225 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 14:40:29 (1758739229) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 14:40:32 (1758739232) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 14:40:35 (1758739235) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 14:40:38 (1758739238) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5261 0x148d 0x280000400 0 5288 0x14a8 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5289 0x14a9 0x240000400 1 5262 0x148e 0x280000400 rc1=0 and rc2=0 PASS 93 (14s) == sanityn test 94: signal vs CP callback race =========== 14:40:52 (1758739252) osc.lustre-OST0000-osc-ffff8d17881f1000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8d179dd36800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d17881f1000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d179dd36800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.201064 s, 40.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.93582 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 323194 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0918174 s, 44.6 kB/s osc.lustre-OST0000-osc-ffff8d17881f1000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d179dd36800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d17881f1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d179dd36800.idle_timeout=debug PASS 94 (14s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 14:41:06 (1758739266) llite.lustre-ffff8d17881f1000.fast_read=0 llite.lustre-ffff8d179dd36800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0513154 s, 40.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8d17881f1000.fast_read=1 llite.lustre-ffff8d179dd36800.fast_read=1 PASS 95a (18s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 14:41:24 (1758739284) llite.lustre-ffff8d17881f1000.fast_read=0 llite.lustre-ffff8d179dd36800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00848831 s, 1.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8d17881f1000.fast_read=1 llite.lustre-ffff8d179dd36800.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 14:41:35 (1758739295) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 14:41:38 (1758739298) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00441267 s, 928 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 14:41:44 (1758739304) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0465141 s, 45.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (6s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 14:41:50 (1758739310) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0557913 s, 37.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (5s) == sanityn test 100e: DoM: read on open and file size ==== 14:41:55 (1758739315) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00468401 s, 2.6 MB/s PASS 100e (6s) == sanityn test 101a: Discard DoM data on unlink ========= 14:42:01 (1758739321) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000795853 s, 5.1 MB/s PASS 101a (5s) == sanityn test 101b: Discard DoM data on rename ========= 14:42:06 (1758739326) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00751206 s, 545 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 14:42:12 (1758739332) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000776155 s, 5.3 MB/s PASS 101c (7s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 14:42:20 (1758739340) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1758739343 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1758739346 Finishing test test23 at 1758739346 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 14:42:33 (1758739353) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 14:42:36 (1758739356) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 14:42:58 (1758739378) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 14:43:01 (1758739381) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0157591 s, 65.0 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508502 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:43:03.000000000 -0400 Modify: 2025-09-24 14:43:03.000000000 -0400 Change: 2025-09-24 14:43:03.000000000 -0400 Birth: 2025-09-24 14:43:03.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272508502 -rw-r--r-- 1 0 0 1758739383 1758739383 1758739383 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508502 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:43:03.000000000 -0400 Modify: 2025-09-24 14:43:03.000000000 -0400 Change: 2025-09-24 14:43:03.000000000 -0400 Birth: 2025-09-24 14:43:03.000000000 -0400 1758739383 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 14:43:08 (1758739388) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 14:43:14 (1758739394) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.195313 s, 53.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5268 0x1494 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 14:43:22 (1758739402) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.235875 s, 44.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5298 0x14b2 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7550 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (12s) == sanityn test 108a: lseek: parallel updates ============ 14:43:34 (1758739414) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00823399 s, 1.2 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.64283 s, 1.8 MB/s PASS 108a (10s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 14:43:44 (1758739424) 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Iteration 25 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Iteration 46 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 192.168.203.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 PASS 109 (816s) == sanityn test 110: do not grant another lock on resend ========================================================== 14:57:20 (1758740240) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-24 14:57:21.000000000 -0400 Modify: 2025-09-24 14:57:21.000000000 -0400 Change: 2025-09-24 14:57:21.000000000 -0400 Birth: 2025-09-24 14:57:21.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg330-client.virtnet oleg330-client.virtnet PASS 110 (66s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 14:58:27 (1758740307) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 14:58:28 (1758740308) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 14:58:30 (1758740310) lustre-OST0000-osc-ffff8d17878a5800 active. lustre-OST0001-osc-ffff8d17878a5800 active. lustre-OST0000-osc-ffff8d17878a5800 active. lustre-OST0001-osc-ffff8d17878a5800 active. MGC192.168.203.130@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 14:58:33 (1758740313) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 14:58:35 (1758740315) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 14:58:36 (1758740316) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 14:58:38 (1758740318) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=20 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1337 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1661 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1594 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1546 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1697 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1809 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1612 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1541 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1792 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2076 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2166 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2012 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2569 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2009 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2444 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2407 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2117 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2196 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2393 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2809 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3694 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3012 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3129 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3044 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3873 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3108 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3135 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.336 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.4321 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.4383 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.5245 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 80.511 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.4838 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.4879 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.474 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.5053 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.507 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.555 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.574 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.593 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d17878a5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d17878a5800.ost_server_uuid in FULL state after 0 sec PASS 200 (109s) cleanup: ====================================================== == sanityn test complete, duration 6773 sec ============== 15:00:28 (1758740428) === sanityn: start cleanup 15:00:29 (1758740429) === Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg330-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 15:02:28 (1758740548) ===