-----============= acceptance-small: sanityn ============----- Thu Jul 24 09:42:49 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg237-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 09:42:58 (1753364578) === Starting client oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Started clients oleg237-client.virtnet: 192.168.202.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg237-client.virtnet: executing check_config_client /mnt/lustre oleg237-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg237-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98bc4509a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff98bc45ca2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98bc4509a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98bc45ca2000.idle_timeout=debug disable quota as required oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 09:43:11 (1753364591) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142739 s, 73.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7541] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 09:43:12 (1753364592) PASS 0 (4s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 09:43:16 (1753364596) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 09:43:19 (1753364599) -rw-r--r-- 1 root root 0 Jul 24 09:43 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 09:43:21 (1753364601) -rw-r--r-- 1 root root 0 Jul 24 09:43 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 09:43:24 (1753364604) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 09:43:27 (1753364607) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 09:43:29 (1753364609) total 1 -rwxrwxrwx 1 root root 0 Jul 24 09:43 f2a -rwxrwxrwx 1 root root 0 Jul 24 09:43 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 24 09:43 f2a -rwxrwxrwx 1 root root 0 Jul 24 09:43 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00460651 s, 111 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 09:43:32 (1753364612) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 09:43:33 (1753364613) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125286 s, 16.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 09:43:36 (1753364616) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 09:43:39 (1753364619) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 09:43:43 (1753364623) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 09:43:46 (1753364626) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 09:43:48 (1753364628) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 09:43:50 (1753364630) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (3s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 09:43:53 (1753364633) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 09:43:55 (1753364635) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190093 s, 55.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0219652 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (3s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 09:43:58 (1753364638) 1+0 records in 1+0 records out 1 byte copied, 0.00213671 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00375741 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00409818 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0059661 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00418913 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00543125 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00751151 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00642911 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113644 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00595074 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112483 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00456306 s, 0.2 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 09:44:01 (1753364641) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0041741 s, 736 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00934087 s, 439 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000178354 s, 17.2 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 09:44:03 (1753364643) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 09:44:06 (1753364646) warning: '-runtime' deprecated, use '-t runtime' instead - link 1691 (time 1753364657.34 total 10.01 last 168.94) - link 2850 (time 1753364667.34 total 20.01 last 115.89) - link 4379 (time 1753364677.35 total 30.02 last 152.77) - link 5660 (time 1753364687.35 total 40.02 last 128.08) - link 6758 (time 1753364697.36 total 50.03 last 109.73) - link 7823 (time 1753364707.36 total 60.03 last 106.43) - link 8909 (time 1753364717.37 total 70.04 last 108.56) - link 9939 (time 1753364727.37 total 80.04 last 102.96) - link 11106 (time 1753364737.37 total 90.04 last 116.67) total: 12246 link in 100.01 seconds: 122.45 ops/second using seed 3853800218 running for 100 seconds - stat 10000 (time 1753364656 ; total 8 ; last 8) - stat 20000 (time 1753364664 ; total 16 ; last 8) - stat 30000 (time 1753364671 ; total 23 ; last 7) - stat 40000 (time 1753364679 ; total 31 ; last 8) - stat 50000 (time 1753364687 ; total 39 ; last 8) - stat 60000 (time 1753364694 ; total 46 ; last 7) - stat 67712 (time 1753364705 ; total 57 ; last 11) - stat 70000 (time 1753364709 ; total 61 ; last 4) - stat 75981 (time 1753364720 ; total 72 ; last 11) - stat 80000 (time 1753364727 ; total 79 ; last 7) - stat 85824 (time 1753364738 ; total 90 ; last 11) - stat 90000 (time 1753364745 ; total 97 ; last 7) total: 91946 stats in 100 seconds: 919.460022 stats/second - unlinked 0 (time 1753364698 ; total 0 ; last 0) - unlinked 10000 (time 1753364795 ; total 97 ; last 97) unlink(/mnt/lustre2/lockdir/lockfile12246) error: No such file or directory total: 12245 unlinks in 109 seconds: 112.339447 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21315) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21317) - No such process PASS 12 (164s) == sanityn test 13: test directory page revocation ======= 09:46:50 (1753364810) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:46:53 (1753364813) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 22507 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:46:56 (1753364816) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 23086 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:46:59 (1753364819) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23663 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:47:02 (1753364822) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24245 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:47:04 (1753364824) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24826 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:47:08 (1753364828) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7524352 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7524352KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (32s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:47:40 (1753364860) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.250174 s, 33.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.189994 s, 44.2 MB/s Chance of close/open is 1 in 50 Seed set to 4861 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1753364862.136619 write 0x46a3c7 thru 0x470d7b (0x69b5 bytes) 000005[1] 1753364862.168482 read 0x02defe thru 0x036893 (0x8996 bytes) 000009[0] 1753364862.239557 mapread 0x2c3031 thru 0x2ce9d8 (0xb9a8 bytes) truncating to largest ever: 0x1c10ff 000013[0] 1753364862.281226 trunc from 0x470d7c to 0x1c10fe (0x2afc7d bytes) 000014[0] 1753364862.309261 mapwrite 0x133ed2 thru 0x13ca76 (0x8ba5 bytes) 000016[1] 1753364862.398914 mapread 0x140335 thru 0x146a44 (0x6710 bytes) 000017[1] 1753364862.431912 read 0x160ea5 thru 0x163b5c (0x2cb8 bytes) 000018[0] 1753364862.435741 mapwrite 0x03d68e thru 0x03f246 (0x1bb9 bytes) 000019[0] 1753364862.488696 mapread 0x0024e1 thru 0x0103a5 (0xdec5 bytes) 000021[0] 1753364862.534635 mapread 0x182218 thru 0x18b690 (0x9479 bytes) 000022[1] 1753364862.546599 write 0x5f1a84 thru 0x5f8de5 (0x7362 bytes) 000024[1] 1753364862.574155 mapwrite 0x0e1740 thru 0x0ebb25 (0xa3e6 bytes) 000025[0] 1753364862.644374 write 0x7f16e9 thru 0x7f5169 (0x3a81 bytes) 000026[0] 1753364862.673485 trunc from 0x7f516a to 0x191281 (0x663ee8 bytes) 000027[1] 1753364862.716891 trunc from 0x191282 to 0x0a6820 (0xeaa61 bytes) 000028[1] 1753364862.744076 read 0x003f25 thru 0x009e93 (0x5f6f bytes) 000029[1] 1753364862.762223 mapwrite 0x39b389 thru 0x3a9f6b (0xebe3 bytes) 000030[1] 1753364862.860351 mapread 0x07b69d thru 0x07e2f4 (0x2c58 bytes) 000032[1] 1753364862.881727 write 0x6327d7 thru 0x63c796 (0x9fc0 bytes) 000034[0] 1753364862.883818 write 0x7addf5 thru 0x7bc6c8 (0xe8d4 bytes) 000035[0] 1753364862.906190 write 0x207a3e thru 0x20eaa1 (0x7064 bytes) 000037[0] 1753364862.938202 read 0x30308c thru 0x311717 (0xe68c bytes) 000041[0] 1753364862.955299 mapwrite 0x78f444 thru 0x794515 (0x50d2 bytes) 000043[1] 1753364862.992496 write 0x4c20f6 thru 0x4cfa43 (0xd94e bytes) 000048[0] 1753364863.041096 write 0x8cc764 thru 0x8d3e1d (0x76ba bytes) 000050[0] 1753364863.054403 mapread 0x1c03d6 thru 0x1c44f5 (0x4120 bytes) 000055[0] 1753364863.104959 read 0x561c2d thru 0x56b88c (0x9c60 bytes) 000058[1] 1753364863.124145 read 0x74f212 thru 0x7585eb (0x93da bytes) 000059[1] 1753364863.128954 read 0x5c53b5 thru 0x5d33ec (0xe038 bytes) 000060[1] 1753364863.137597 mapwrite 0x77ed3a thru 0x78e8e0 (0xfba7 bytes) truncating to largest ever: 0x4a9b98 000062[0] 1753364863.181344 trunc from 0x8d3e1e to 0x4a9b97 (0x42a286 bytes) 000065[1] 1753364863.208933 write 0x03258d thru 0x03bac5 (0x9539 bytes) 000066[1] 1753364863.225778 mapwrite 0x6a604a thru 0x6a8556 (0x250d bytes) 000067[0] 1753364863.274251 trunc from 0x6a8557 to 0x21c460 (0x48c0f6 bytes) 000068[1] 1753364863.295287 read 0x18456c thru 0x19174f (0xd1e4 bytes) 000069[1] 1753364863.325277 read 0x06617e thru 0x074a14 (0xe897 bytes) 000070[1] 1753364863.325877 read 0x00106f thru 0x00f828 (0xe7ba bytes) 000072[0] 1753364863.338919 mapwrite 0x3e25dc thru 0x3e7b90 (0x55b5 bytes) 000076[1] 1753364863.394682 mapread 0x33b2e2 thru 0x345e37 (0xab56 bytes) 000077[1] 1753364863.444325 write 0x29a6e0 thru 0x2a6300 (0xbc21 bytes) 000078[1] 1753364863.460000 mapread 0x2a52d2 thru 0x2ab54c (0x627b bytes) 000079[0] 1753364863.474952 mapwrite 0x430bbe thru 0x43fbfa (0xf03d bytes) truncating to largest ever: 0x53fb8b 000080[1] 1753364863.520262 trunc from 0x43fbfb to 0x53fb8a (0xfff90 bytes) 000083[0] 1753364863.543411 trunc from 0x53fb8b to 0x29f147 (0x2a0a43 bytes) 000084[0] 1753364863.562407 mapread 0x1d2934 thru 0x1d84a9 (0x5b76 bytes) 000085[0] 1753364863.577998 trunc from 0x29f148 to 0x2a3504 (0x43bd bytes) truncating to largest ever: 0x90a5d5 000088[1] 1753364863.585182 trunc from 0x2a3505 to 0x90a5d4 (0x6670d0 bytes) truncating to largest ever: 0x9be2e9 000090[1] 1753364863.602769 trunc from 0x90a5d5 to 0x9be2e8 (0xb3d14 bytes) 000093[1] 1753364863.614415 read 0x5a959a thru 0x5b5c1c (0xc683 bytes) 000094[0] 1753364863.621818 write 0x15c14e thru 0x169ea2 (0xdd55 bytes) 000097[0] 1753364863.640157 read 0x2bd3c6 thru 0x2cb300 (0xdf3b bytes) 000098[0] 1753364863.649022 mapread 0x86e628 thru 0x871817 (0x31f0 bytes) 000099[1] 1753364863.667524 trunc from 0x9be2e9 to 0x31fc96 (0x69e652 bytes) 000100[0] 1753364863.690548 mapread 0x18b491 thru 0x18b7d2 (0x342 bytes) 000102[0] 1753364863.709721 mapwrite 0x04d111 thru 0x04d2b1 (0x1a1 bytes) 000103[1] 1753364863.741648 write 0x1ac62b thru 0x1baff6 (0xe9cc bytes) 000107[0] 1753364863.761165 write 0x9f5142 thru 0x9fffff (0xaebe bytes) 000109[0] 1753364863.782480 write 0x43fe96 thru 0x449f98 (0xa103 bytes) 000111[0] 1753364863.797665 read 0x27b9f0 thru 0x28b3a3 (0xf9b4 bytes) 000112[1] 1753364863.808464 mapread 0x4e68d4 thru 0x4ea517 (0x3c44 bytes) 000113[1] 1753364863.847282 mapwrite 0x3aa28c thru 0x3b8b35 (0xe8aa bytes) 000114[1] 1753364863.890985 write 0x409a24 thru 0x412fcb (0x95a8 bytes) 000115[1] 1753364863.905662 mapread 0x5f2658 thru 0x5ff88d (0xd236 bytes) 000117[1] 1753364863.939285 trunc from 0xa00000 to 0x29d44c (0x762bb3 bytes) 000119[1] 1753364863.953669 read 0x025eb8 thru 0x02604c (0x195 bytes) 000121[0] 1753364863.982789 mapread 0x0883ae thru 0x091ba5 (0x97f8 bytes) 000125[1] 1753364864.013625 trunc from 0x29d44d to 0x767536 (0x4ca0ea bytes) 000126[1] 1753364864.031544 mapwrite 0x55f04e thru 0x5684d2 (0x9485 bytes) 000127[1] 1753364864.086944 mapread 0x642e85 thru 0x6448f2 (0x1a6e bytes) 000129[1] 1753364864.087852 mapwrite 0x931d04 thru 0x9326e2 (0x9df bytes) 000132[0] 1753364864.145749 trunc from 0x9326e3 to 0x2cb49b (0x667247 bytes) 000137[0] 1753364864.188002 write 0x8c81f7 thru 0x8d1fb7 (0x9dc1 bytes) 000138[0] 1753364864.190668 mapread 0x536914 thru 0x539c47 (0x3334 bytes) 000139[0] 1753364864.198560 mapwrite 0x10a676 thru 0x10ba21 (0x13ac bytes) 000140[0] 1753364864.220341 trunc from 0x8d1fb8 to 0x6bb13b (0x216e7c bytes) 000145[0] 1753364864.237137 mapread 0x5b31e4 thru 0x5bc7af (0x95cc bytes) 000146[0] 1753364864.244106 mapread 0x668aa7 thru 0x674652 (0xbbac bytes) 000149[1] 1753364864.264598 read 0x40adb1 thru 0x40f4ae (0x46fe bytes) 000150[0] 1753364864.276648 write 0x394e35 thru 0x39c6d6 (0x78a2 bytes) 000151[0] 1753364864.286510 mapread 0x676f78 thru 0x682884 (0xb90d bytes) 000154[0] 1753364864.305238 mapread 0x2238b5 thru 0x229751 (0x5e9d bytes) 000155[1] 1753364864.316988 write 0x3a3267 thru 0x3ab351 (0x80eb bytes) 000156[0] 1753364864.334683 write 0x158cc4 thru 0x162d72 (0xa0af bytes) 000157[0] 1753364864.348103 mapwrite 0x1bed02 thru 0x1cc379 (0xd678 bytes) 000158[0] 1753364864.382504 write 0x71dd58 thru 0x72c21f (0xe4c8 bytes) 000159[0] 1753364864.391510 trunc from 0x72c220 to 0x78bf65 (0x5fd46 bytes) 000163[1] 1753364864.398682 write 0x3c90ca thru 0x3d3d9a (0xacd1 bytes) 000167[0] 1753364864.425990 mapwrite 0x2f5d74 thru 0x2ff0d5 (0x9362 bytes) 000168[1] 1753364864.470064 mapread 0x2e5a6e thru 0x2ecf77 (0x750a bytes) 000170[0] 1753364864.483517 read 0x149e0b thru 0x153af7 (0x9ced bytes) 000172[1] 1753364864.497557 trunc from 0x78bf66 to 0x619639 (0x17292c bytes) 000173[1] 1753364864.512740 trunc from 0x61963a to 0x4abe62 (0x16d7d7 bytes) 000174[0] 1753364864.524857 mapread 0x1ec0e8 thru 0x1f0f6b (0x4e84 bytes) 000175[0] 1753364864.542503 mapread 0x19e18d thru 0x1ab6bf (0xd533 bytes) 000176[0] 1753364864.557355 write 0x5770a8 thru 0x582907 (0xb860 bytes) 000178[0] 1753364864.569867 write 0x74d8dc thru 0x74f327 (0x1a4c bytes) 000179[0] 1753364864.577255 mapread 0x3d20d9 thru 0x3db6fb (0x9623 bytes) 000180[1] 1753364864.587062 trunc from 0x74f328 to 0x6697e0 (0xe5b47 bytes) 000182[0] 1753364864.614533 mapwrite 0x1a6510 thru 0x1a6fb7 (0xaa8 bytes) 000184[0] 1753364864.644482 write 0x5a4fed thru 0x5ae431 (0x9445 bytes) 000189[0] 1753364864.663936 write 0x2f9d10 thru 0x2fadc2 (0x10b3 bytes) 000190[1] 1753364864.672415 read 0x1d80dd thru 0x1e0299 (0x81bd bytes) 000191[0] 1753364864.682584 mapread 0x159b69 thru 0x167a39 (0xded1 bytes) 000193[0] 1753364864.692133 read 0x644d83 thru 0x64bd6e (0x6fec bytes) 000197[1] 1753364864.709539 mapwrite 0x3772e1 thru 0x384c8e (0xd9ae bytes) 000201[1] 1753364864.765123 write 0x3b57c9 thru 0x3baddd (0x5615 bytes) 000202[1] 1753364864.770395 write 0x53e678 thru 0x54a1f4 (0xbb7d bytes) 000203[0] 1753364864.790771 mapwrite 0x1723c3 thru 0x177ba1 (0x57df bytes) 000205[0] 1753364864.831948 write 0x9372bd thru 0x93ef66 (0x7caa bytes) 000209[1] 1753364864.852509 read 0x936e36 thru 0x93ef66 (0x8131 bytes) 000210[0] 1753364864.868971 write 0x959b1b thru 0x95af0c (0x13f2 bytes) 000212[0] 1753364864.882455 mapwrite 0x23ebe6 thru 0x24343c (0x4857 bytes) 000213[1] 1753364864.908018 mapread 0x58585f thru 0x5859aa (0x14c bytes) 000214[0] 1753364864.918394 mapwrite 0x23fbf3 thru 0x247384 (0x7792 bytes) 000217[1] 1753364864.956659 read 0x51fb7b thru 0x528084 (0x850a bytes) 000218[0] 1753364864.963710 read 0x245dca thru 0x24c72e (0x6965 bytes) 000219[0] 1753364864.968004 read 0x151841 thru 0x15fd03 (0xe4c3 bytes) 000220[0] 1753364864.975311 write 0x8f4790 thru 0x8f5a19 (0x128a bytes) 000223[1] 1753364864.992297 mapread 0x27cde0 thru 0x282186 (0x53a7 bytes) 000225[1] 1753364865.010056 read 0x7eabb4 thru 0x7f3a3d (0x8e8a bytes) 000229[1] 1753364865.023776 mapread 0x82759d thru 0x836ab2 (0xf516 bytes) 000231[0] 1753364865.033535 write 0x9238d9 thru 0x925649 (0x1d71 bytes) 000232[0] 1753364865.048555 read 0x0aa8f3 thru 0x0b983b (0xef49 bytes) 000233[1] 1753364865.056849 write 0x16deb3 thru 0x1789b1 (0xaaff bytes) 000235[0] 1753364865.076660 mapread 0x469019 thru 0x471d91 (0x8d79 bytes) 000236[0] 1753364865.102705 write 0x6b7a07 thru 0x6b856c (0xb66 bytes) 000237[0] 1753364865.114740 read 0x0cdb2b thru 0x0cfeb6 (0x238c bytes) 000241[1] 1753364865.135102 write 0x44a773 thru 0x456342 (0xbbd0 bytes) 000243[1] 1753364865.148556 read 0x7d6bdf thru 0x7d6fd4 (0x3f6 bytes) 000244[0] 1753364865.150878 trunc from 0x95af0d to 0x445439 (0x515ad3 bytes) 000246[0] 1753364865.176443 trunc from 0x44543a to 0x44357b (0x1ebe bytes) 000248[1] 1753364865.196523 write 0x80b7c7 thru 0x8151d9 (0x9a13 bytes) 000249[1] 1753364865.207059 write 0x14b688 thru 0x1549cd (0x9346 bytes) 000250[1] 1753364865.217115 write 0x8ddb13 thru 0x8e97f0 (0xbcde bytes) 000251[1] 1753364865.225644 read 0x0864a8 thru 0x08b8cf (0x5428 bytes) 000252[1] 1753364865.238967 write 0x6d0131 thru 0x6dec09 (0xead9 bytes) 000255[0] 1753364865.262762 mapwrite 0x2732b4 thru 0x276776 (0x34c3 bytes) 000258[0] 1753364865.320085 write 0x0057a1 thru 0x0129f0 (0xd250 bytes) 000259[0] 1753364865.335343 mapread 0x23f8f2 thru 0x245b02 (0x6211 bytes) 000262[1] 1753364865.365165 mapread 0x159e94 thru 0x1695c9 (0xf736 bytes) 000263[1] 1753364865.394037 trunc from 0x8e97f1 to 0x0c3953 (0x825e9d bytes) 000264[0] 1753364865.415785 read 0x07f671 thru 0x0846a2 (0x5032 bytes) 000265[1] 1753364865.434556 trunc from 0x0c3954 to 0x10942d (0x45ada bytes) 000267[1] 1753364865.455448 read 0x0b8e88 thru 0x0c2d4a (0x9ec3 bytes) 000271[0] 1753364865.482390 read 0x0dcea5 thru 0x0ec894 (0xf9f0 bytes) 000272[1] 1753364865.504403 trunc from 0x10942e to 0x986a20 (0x87d5f3 bytes) 000273[0] 1753364865.519996 mapread 0x03cb16 thru 0x045df5 (0x92e0 bytes) 000274[1] 1753364865.572621 write 0x068e42 thru 0x069b4f (0xd0e bytes) 000275[0] 1753364865.614759 trunc from 0x986a21 to 0x816517 (0x170509 bytes) 000279[1] 1753364865.654658 write 0x3103dd thru 0x314c62 (0x4886 bytes) 000280[1] 1753364865.671417 trunc from 0x816518 to 0x158cf0 (0x6bd827 bytes) 000282[0] 1753364865.695467 mapwrite 0x71a56f thru 0x728664 (0xe0f6 bytes) 000283[1] 1753364865.774171 write 0x627cfb thru 0x62b9a4 (0x3caa bytes) 000284[1] 1753364865.829172 read 0x6f0517 thru 0x6fd7f4 (0xd2de bytes) 000285[1] 1753364865.841629 write 0x9aab1a thru 0x9b9163 (0xe64a bytes) 000287[0] 1753364865.848418 mapread 0x461de5 thru 0x470b7a (0xed96 bytes) 000288[0] 1753364865.923089 mapread 0x69853a thru 0x69a063 (0x1b2a bytes) 000289[0] 1753364865.945653 mapwrite 0x920e61 thru 0x92705f (0x61ff bytes) 000293[0] 1753364866.030630 read 0x650c1d thru 0x6596ed (0x8ad1 bytes) 000294[0] 1753364866.036445 mapwrite 0x8a4052 thru 0x8a6933 (0x28e2 bytes) 000297[0] 1753364866.076040 mapwrite 0x35b534 thru 0x367a46 (0xc513 bytes) 000298[0] 1753364866.132286 trunc from 0x9b9164 to 0x5f68c2 (0x3c28a1 bytes) 000302[1] 1753364866.186451 write 0x1f684f thru 0x200b68 (0xa31a bytes) 000303[1] 1753364866.204071 read 0x504c3e thru 0x50b731 (0x6af4 bytes) 000305[0] 1753364866.236163 read 0x254b64 thru 0x25dfc3 (0x9460 bytes) 000306[1] 1753364866.253893 mapread 0x09fbaf thru 0x0aef20 (0xf372 bytes) 000307[1] 1753364866.270541 mapwrite 0x67526d thru 0x681402 (0xc196 bytes) 000314[1] 1753364866.363141 mapread 0x1a9c7c thru 0x1adb27 (0x3eac bytes) 000316[1] 1753364866.380184 mapread 0x43d243 thru 0x443cbf (0x6a7d bytes) 000318[1] 1753364866.394357 mapread 0x1a3855 thru 0x1ae67c (0xae28 bytes) 000320[1] 1753364866.403721 read 0x3bf2d9 thru 0x3bf52b (0x253 bytes) 000322[0] 1753364866.420109 trunc from 0x681403 to 0x972429 (0x2f1027 bytes) 000324[0] 1753364866.455521 mapread 0x80c30b thru 0x81483e (0x8534 bytes) 000326[1] 1753364866.476026 read 0x3b8416 thru 0x3c5781 (0xd36c bytes) 000327[1] 1753364866.489927 read 0x3a8800 thru 0x3aed2b (0x652c bytes) 000330[0] 1753364866.502231 write 0x7d8be7 thru 0x7e2567 (0x9981 bytes) 000331[1] 1753364866.509448 read 0x5494ee thru 0x54d860 (0x4373 bytes) 000332[0] 1753364866.524828 mapread 0x4a330d thru 0x4a7554 (0x4248 bytes) 000333[1] 1753364866.537576 read 0x16fb35 thru 0x172786 (0x2c52 bytes) 000334[1] 1753364866.542281 mapwrite 0x5a35d5 thru 0x5aa22f (0x6c5b bytes) 000336[0] 1753364866.611695 mapread 0x390b3d thru 0x399062 (0x8526 bytes) 000338[0] 1753364866.643551 trunc from 0x97242a to 0x452619 (0x51fe10 bytes) 000339[0] 1753364866.675534 mapread 0x40b46c thru 0x416efc (0xba91 bytes) 000341[0] 1753364866.705775 mapread 0x25b946 thru 0x268905 (0xcfc0 bytes) 000342[0] 1753364866.723847 mapread 0x394f99 thru 0x39d91f (0x8987 bytes) 000343[1] 1753364866.734614 mapwrite 0x7574b6 thru 0x759477 (0x1fc2 bytes) 000345[0] 1753364866.803302 read 0x61292e thru 0x61a94c (0x801f bytes) 000346[1] 1753364866.827644 read 0x68f8e6 thru 0x697fe8 (0x8703 bytes) 000347[1] 1753364866.855563 trunc from 0x759478 to 0x454d11 (0x304766 bytes) 000349[0] 1753364866.872555 trunc from 0x454d12 to 0x56170d (0x10c9fc bytes) 000353[0] 1753364866.919574 read 0x532a0e thru 0x541918 (0xef0b bytes) 000354[1] 1753364866.925154 write 0x3f43b7 thru 0x3f8d8a (0x49d4 bytes) 000356[1] 1753364866.960132 trunc from 0x56170e to 0x2a60bb (0x2bb652 bytes) 000359[0] 1753364867.001279 trunc from 0x2a60bc to 0x792d45 (0x4ecc8a bytes) 000365[1] 1753364867.051369 mapread 0x66d3ea thru 0x67457c (0x7193 bytes) 000366[1] 1753364867.076495 write 0x060954 thru 0x065aaa (0x5157 bytes) 000371[0] 1753364867.095487 read 0x5e1fc1 thru 0x5ef881 (0xd8c1 bytes) 000374[1] 1753364867.110337 trunc from 0x792d46 to 0x112d66 (0x67ffdf bytes) 000377[1] 1753364867.131570 mapwrite 0x055745 thru 0x056e45 (0x1701 bytes) 000378[0] 1753364867.159931 trunc from 0x112d67 to 0x40efff (0x2fc299 bytes) 000380[1] 1753364867.196138 write 0x655187 thru 0x65d880 (0x86fa bytes) 000382[1] 1753364867.216180 read 0x5c77cc thru 0x5d49c1 (0xd1f6 bytes) 000385[1] 1753364867.242109 mapwrite 0x5d2481 thru 0x5e188a (0xf40a bytes) 000386[0] 1753364867.301997 write 0x989983 thru 0x98b495 (0x1b13 bytes) 000387[1] 1753364867.318362 mapread 0x6a3b26 thru 0x6a3ef2 (0x3cd bytes) 000388[1] 1753364867.346300 mapwrite 0x151536 thru 0x15e131 (0xcbfc bytes) 000390[1] 1753364867.428404 read 0x51d49a thru 0x526d22 (0x9889 bytes) 000393[0] 1753364867.461471 mapwrite 0x8e85d9 thru 0x8e9f6f (0x1997 bytes) 000395[1] 1753364867.522603 trunc from 0x98b496 to 0x6433aa (0x3480eb bytes) 000396[0] 1753364867.545676 mapread 0x3f2121 thru 0x40189f (0xf77f bytes) 000397[0] 1753364867.577088 mapread 0x1ff11e thru 0x204220 (0x5103 bytes) 000398[0] 1753364867.590189 read 0x1c241f thru 0x1c5b9e (0x3780 bytes) 000401[0] 1753364867.598992 mapread 0x0f7264 thru 0x1003bd (0x915a bytes) 000402[1] 1753364867.608756 mapwrite 0x02cb2c thru 0x0301d0 (0x36a5 bytes) 000403[1] 1753364867.656112 write 0x4562ff thru 0x4661a8 (0xfeaa bytes) 000404[0] 1753364867.667051 mapread 0x3a4b7a thru 0x3abcf8 (0x717f bytes) 000405[1] 1753364867.696896 read 0x1f73c5 thru 0x1fce7f (0x5abb bytes) 000409[1] 1753364867.724694 mapwrite 0x48203b thru 0x482641 (0x607 bytes) 000410[0] 1753364867.775632 mapwrite 0x767e5b thru 0x76a07d (0x2223 bytes) 000412[0] 1753364867.890383 mapwrite 0x9a2cae thru 0x9ac02a (0x937d bytes) 000416[0] 1753364867.994269 read 0x02e72b thru 0x03ac17 (0xc4ed bytes) 000417[1] 1753364868.010431 mapwrite 0x4b297e thru 0x4bb3fa (0x8a7d bytes) 000419[1] 1753364868.113181 trunc from 0x9ac02b to 0x6cf4b1 (0x2dcb79 bytes) 000421[0] 1753364868.151935 mapwrite 0x90331f thru 0x90eb0e (0xb7f0 bytes) 000423[1] 1753364868.236829 write 0x2d1db2 thru 0x2e0526 (0xe775 bytes) 000428[0] 1753364868.327101 read 0x77253d thru 0x77d4cd (0xaf91 bytes) 000431[0] 1753364868.362894 read 0x6a476c thru 0x6a6671 (0x1f06 bytes) 000432[0] 1753364868.373976 mapread 0x4fb81f thru 0x506b2d (0xb30f bytes) 000433[1] 1753364868.401921 mapread 0x190d94 thru 0x19654a (0x57b7 bytes) 000434[1] 1753364868.425022 mapread 0x61a8a9 thru 0x61f311 (0x4a69 bytes) 000435[0] 1753364868.461059 write 0x266317 thru 0x266db6 (0xaa0 bytes) 000436[1] 1753364868.485376 mapread 0x48ca55 thru 0x49a97d (0xdf29 bytes) 000438[0] 1753364868.503387 mapread 0x671594 thru 0x67598b (0x43f8 bytes) 000441[1] 1753364868.522061 mapwrite 0x6ff255 thru 0x706f49 (0x7cf5 bytes) 000442[1] 1753364868.564154 mapread 0x454f47 thru 0x463250 (0xe30a bytes) 000445[1] 1753364868.611357 mapwrite 0x1f7615 thru 0x1fdb70 (0x655c bytes) 000447[0] 1753364868.660506 read 0x7bebad thru 0x7c2ecb (0x431f bytes) 000448[1] 1753364868.674982 read 0x247ade thru 0x24cd0c (0x522f bytes) 000449[1] 1753364868.680522 read 0x65090a thru 0x65ed94 (0xe48b bytes) 000456[0] 1753364868.716470 write 0x3676ef thru 0x370886 (0x9198 bytes) 000464[0] 1753364868.752035 mapwrite 0x482df3 thru 0x48b4bb (0x86c9 bytes) 000465[0] 1753364868.800830 write 0x3bec09 thru 0x3c3b4a (0x4f42 bytes) 000467[0] 1753364868.809622 read 0x30d21b thru 0x30fde8 (0x2bce bytes) 000468[1] 1753364868.824064 mapread 0x034da9 thru 0x03f984 (0xabdc bytes) 000469[1] 1753364868.860935 mapread 0x06a97a thru 0x06c15b (0x17e2 bytes) 000470[0] 1753364868.888204 write 0x75684e thru 0x76143f (0xabf2 bytes) 000472[1] 1753364868.906094 read 0x6bd60a thru 0x6c1824 (0x421b bytes) 000476[1] 1753364868.928891 read 0x18f411 thru 0x192f60 (0x3b50 bytes) 000478[1] 1753364868.933141 read 0x415ef4 thru 0x4184d6 (0x25e3 bytes) 000479[1] 1753364868.938513 write 0x6fb741 thru 0x6fbdfa (0x6ba bytes) 000480[0] 1753364868.948691 mapread 0x3aaf62 thru 0x3ab256 (0x2f5 bytes) 000482[0] 1753364868.969004 mapwrite 0x4a1930 thru 0x4a9c23 (0x82f4 bytes) 000484[1] 1753364869.039369 mapread 0x1526b2 thru 0x160c3b (0xe58a bytes) 000487[1] 1753364869.081351 mapwrite 0x788682 thru 0x7946d3 (0xc052 bytes) truncating to largest ever: 0x9f17ad 000492[0] 1753364869.168342 trunc from 0x90eb0f to 0x9f17ac (0xe2c9e bytes) 000493[0] 1753364869.191909 mapwrite 0x33a3a8 thru 0x33ddf6 (0x3a4f bytes) 000494[1] 1753364869.255510 trunc from 0x9f17ad to 0x7f53fd (0x1fc3af bytes) 000496[0] 1753364869.289568 mapread 0x1db4c2 thru 0x1dbc97 (0x7d6 bytes) 000497[0] 1753364869.305275 read 0x352255 thru 0x3558c4 (0x3670 bytes) 000500[0] 1753364869.324534 mapread 0x206194 thru 0x20a8a1 (0x470e bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4869 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0xddab5 000001[0] 1753364869.633549 trunc from 00000000 to 0x0ddab4 (0xddab5 bytes) 000002[0] 1753364869.637773 write 0x4d0000 thru 0x4d445a (0x445b bytes) 000003[0] 1753364869.644314 read 0x2b1000 thru 0x2bbf41 (0xaf42 bytes) 000004[1] 1753364869.647508 read_OD 0x3fd000 thru 0x409fff (0xd000 bytes) 000005[0] 1753364869.657572 write 0x8f6000 thru 0x8ff1c7 (0x91c8 bytes) 000006[1] 1753364869.666150 read_OD 0x00b000 thru 0x00bfff (0x1000 bytes) truncating to largest ever: 0x453a92 000007[1] 1753364869.675722 trunc from 0x8ff1c8 to 0x453a91 (0x4ab736 bytes) truncating to largest ever: 0x77a392 000009[1] 1753364869.693064 trunc from 0x453a92 to 0x77a391 (0x326900 bytes) 000010[1] 1753364869.708133 mapwrite 0x2ec000 thru 0x2f4e21 (0x8e22 bytes) 000011[0] 1753364869.752411 write 0x5aa000 thru 0x5ad3fd (0x33fe bytes) 000016[0] 1753364869.768520 mapwrite 0x3c6000 thru 0x3c79dd (0x19de bytes) 000017[1] 1753364869.790374 write_OD 0x61d000 thru 0x61dfff (0x1000 bytes) 000019[1] 1753364869.807844 mapwrite 0x648000 thru 0x648be4 (0xbe5 bytes) 000020[0] 1753364869.825743 mapwrite 0x061000 thru 0x06cc60 (0xbc61 bytes) 000021[0] 1753364869.855599 trunc from 0x77a392 to 0x4e4c1e (0x295773 bytes) 000022[1] 1753364869.868117 write_OD 0x25c000 thru 0x267fff (0xc000 bytes) 000024[1] 1753364869.889531 mapread 0x0ed000 thru 0x0f266a (0x566b bytes) 000027[1] 1753364869.903470 trunc from 0x4e4c1f to 0x18c067 (0x358bb7 bytes) 000030[1] 1753364869.932748 write_OD 0x2db000 thru 0x2e2fff (0x8000 bytes) 000031[0] 1753364869.950389 mapread 0x19a000 thru 0x1a1f2c (0x7f2d bytes) 000036[1] 1753364869.958077 mapwrite 0x07c000 thru 0x07f09a (0x309b bytes) 000037[1] 1753364869.981894 mapread 0x073000 thru 0x082e0b (0xfe0c bytes) 000038[0] 1753364869.988882 read 0x2c2000 thru 0x2d0073 (0xe074 bytes) 000039[1] 1753364870.011614 write_OD 0x8f2000 thru 0x8f3fff (0x2000 bytes) 000042[1] 1753364870.035235 write_OD 0x906000 thru 0x90afff (0x5000 bytes) 000043[0] 1753364870.052242 mapwrite 0x254000 thru 0x25ddfc (0x9dfd bytes) 000044[1] 1753364870.102684 mapread 0x6fb000 thru 0x70538b (0xa38c bytes) 000045[1] 1753364870.119656 mapwrite 0x2ca000 thru 0x2cf06b (0x506c bytes) 000049[1] 1753364870.155994 read_OD 0x763000 thru 0x76efff (0xc000 bytes) 000053[1] 1753364870.164035 mapwrite 0x651000 thru 0x6533a3 (0x23a4 bytes) 000054[0] 1753364870.195262 read 0x241000 thru 0x241e46 (0xe47 bytes) 000055[1] 1753364870.199981 read_OD 0x510000 thru 0x519fff (0xa000 bytes) 000056[0] 1753364870.203680 mapread 0x888000 thru 0x8907ff (0x8800 bytes) 000059[1] 1753364870.210024 trunc from 0x90b000 to 0x6fe413 (0x20cbec bytes) 000061[1] 1753364870.228855 mapwrite 0x074000 thru 0x07a670 (0x6671 bytes) 000062[1] 1753364870.254383 mapwrite 0x2fa000 thru 0x3004da (0x64db bytes) 000063[1] 1753364870.283495 trunc from 0x6fe414 to 0x27fd98 (0x47e67b bytes) 000064[0] 1753364870.292175 mapwrite 0x15b000 thru 0x1672fe (0xc2ff bytes) 000066[0] 1753364870.328967 write 0x9b1000 thru 0x9b3f3f (0x2f40 bytes) 000067[0] 1753364870.331269 mapwrite 0x9fa000 thru 0x9ff0c7 (0x50c8 bytes) 000068[0] 1753364870.370147 mapwrite 0x225000 thru 0x225865 (0x866 bytes) truncating to largest ever: 0x8b0ca2 000069[1] 1753364870.388308 trunc from 0x9ff0c8 to 0x8b0ca1 (0x14e426 bytes) 000073[1] 1753364870.419925 mapread 0x2b2000 thru 0x2b5c7e (0x3c7f bytes) 000075[0] 1753364870.423871 mapread 0x7e2000 thru 0x7ecead (0xaeae bytes) 000078[0] 1753364870.438318 mapread 0x52d000 thru 0x535331 (0x8332 bytes) 000079[0] 1753364870.444673 mapread 0x212000 thru 0x2201a6 (0xe1a7 bytes) 000081[1] 1753364870.453168 mapwrite 0x085000 thru 0x0936f2 (0xe6f3 bytes) 000082[0] 1753364870.482349 write 0x38c000 thru 0x391e51 (0x5e52 bytes) 000083[1] 1753364870.490793 mapread 0x322000 thru 0x32901f (0x7020 bytes) 000084[0] 1753364870.503275 mapread 0x65e000 thru 0x66b8fb (0xd8fc bytes) 000085[1] 1753364870.517596 mapread 0x348000 thru 0x34e78c (0x678d bytes) 000086[1] 1753364870.522063 mapread 0x0bf000 thru 0x0c13e7 (0x23e8 bytes) 000087[0] 1753364870.529334 mapwrite 0x4c5000 thru 0x4cc8b1 (0x78b2 bytes) 000088[1] 1753364870.556996 mapwrite 0x78b000 thru 0x78bf1a (0xf1b bytes) 000089[0] 1753364870.576976 read 0x58b000 thru 0x593adc (0x8add bytes) 000090[1] 1753364870.583736 read_OD 0x423000 thru 0x426fff (0x4000 bytes) 000091[0] 1753364870.585552 mapwrite 0x9e9000 thru 0x9ec35f (0x3360 bytes) 000092[1] 1753364870.611977 mapwrite 0x78b000 thru 0x79520a (0xa20b bytes) 000094[1] 1753364870.638944 read_OD 0x18b000 thru 0x18bfff (0x1000 bytes) 000095[1] 1753364870.643702 write_OD 0x163000 thru 0x16afff (0x8000 bytes) 000096[1] 1753364870.658036 mapread 0x96b000 thru 0x9776fc (0xc6fd bytes) 000097[0] 1753364870.669251 read 0x61a000 thru 0x61c5f8 (0x25f9 bytes) 000099[1] 1753364870.672752 write_OD 0x786000 thru 0x78bfff (0x6000 bytes) 000101[1] 1753364870.690137 mapwrite 0x63d000 thru 0x646132 (0x9133 bytes) 000102[0] 1753364870.714746 read 0x20a000 thru 0x21066a (0x666b bytes) 000106[1] 1753364870.723749 mapread 0x567000 thru 0x576735 (0xf736 bytes) 000109[0] 1753364870.730604 read 0x871000 thru 0x87da60 (0xca61 bytes) 000110[0] 1753364870.734158 write 0x6ce000 thru 0x6d2a7d (0x4a7e bytes) 000111[1] 1753364870.739171 mapread 0x482000 thru 0x484b34 (0x2b35 bytes) 000112[1] 1753364870.759331 mapwrite 0x12f000 thru 0x13d81a (0xe81b bytes) 000115[1] 1753364870.788742 read_OD 0x363000 thru 0x36dfff (0xb000 bytes) 000117[1] 1753364870.794644 mapwrite 0x170000 thru 0x17e177 (0xe178 bytes) 000120[1] 1753364870.823754 mapread 0x516000 thru 0x519e89 (0x3e8a bytes) 000122[0] 1753364870.834889 mapwrite 0x61e000 thru 0x62a41b (0xc41c bytes) 000127[0] 1753364870.882641 read 0x64c000 thru 0x64fe89 (0x3e8a bytes) 000128[0] 1753364870.887941 read 0x6ee000 thru 0x6fa473 (0xc474 bytes) 000129[0] 1753364870.890554 read 0x22c000 thru 0x234c5a (0x8c5b bytes) 000131[0] 1753364870.896730 write 0x08f000 thru 0x09d0c0 (0xe0c1 bytes) 000135[1] 1753364870.911148 read_OD 0x951000 thru 0x957fff (0x7000 bytes) 000137[1] 1753364870.920794 mapread 0x97d000 thru 0x9875fc (0xa5fd bytes) 000140[0] 1753364870.927902 write 0x8b0000 thru 0x8b7594 (0x7595 bytes) 000143[0] 1753364870.938120 write 0x2d4000 thru 0x2d62bd (0x22be bytes) 000144[0] 1753364870.944373 write 0x3d5000 thru 0x3e2f8f (0xdf90 bytes) 000149[1] 1753364870.958095 mapread 0x11c000 thru 0x11d7b3 (0x17b4 bytes) 000152[0] 1753364870.973318 trunc from 0x9ec360 to 0x67364f (0x378d10 bytes) 000153[1] 1753364870.988103 write_OD 0x13b000 thru 0x13efff (0x4000 bytes) truncating to largest ever: 0x9c0c5c 000155[0] 1753364871.006236 trunc from 0x673650 to 0x9c0c5b (0x34d60c bytes) 000156[1] 1753364871.015135 mapread 0x4ec000 thru 0x4f06a3 (0x46a4 bytes) 000157[0] 1753364871.026783 mapread 0x4f7000 thru 0x503987 (0xc988 bytes) 000159[0] 1753364871.037476 mapwrite 0x197000 thru 0x1a20dc (0xb0dd bytes) 000163[0] 1753364871.071479 mapread 0x12f000 thru 0x1309c2 (0x19c3 bytes) 000168[0] 1753364871.097572 mapwrite 0x489000 thru 0x49141f (0x8420 bytes) 000171[0] 1753364871.130920 read 0x18d000 thru 0x197f33 (0xaf34 bytes) 000172[1] 1753364871.137582 mapwrite 0x77e000 thru 0x77ed20 (0xd21 bytes) 000174[0] 1753364871.164195 trunc from 0x9c0c5c to 0x6c13ab (0x2ff8b0 bytes) 000175[0] 1753364871.176205 write 0x87f000 thru 0x883588 (0x4589 bytes) 000176[1] 1753364871.181469 mapwrite 0x8ee000 thru 0x8f6d12 (0x8d13 bytes) 000177[1] 1753364871.215144 trunc from 0x8f6d13 to 0x437e0a (0x4bef08 bytes) 000178[1] 1753364871.225749 read 0x31e000 thru 0x322837 (0x4838 bytes) 000182[0] 1753364871.246684 write 0x3e5000 thru 0x3f27e4 (0xd7e5 bytes) 000184[1] 1753364871.253164 mapread 0x3d1000 thru 0x3e0a55 (0xfa56 bytes) 000185[0] 1753364871.275511 read 0x2bd000 thru 0x2c51b1 (0x81b2 bytes) 000190[1] 1753364871.281794 write 0x978000 thru 0x98737f (0xf380 bytes) 000192[1] 1753364871.296202 trunc from 0x987380 to 0x86db19 (0x119866 bytes) 000194[1] 1753364871.311568 mapwrite 0x796000 thru 0x7a1fde (0xbfdf bytes) 000195[0] 1753364871.338927 trunc from 0x86db1a to 0x8d72b0 (0x69797 bytes) 000197[1] 1753364871.360775 trunc from 0x8d72b1 to 0x5e5b41 (0x2f176f bytes) 000198[1] 1753364871.370864 trunc from 0x5e5b42 to 0x51b2a0 (0xca8a1 bytes) 000199[0] 1753364871.379775 write 0x050000 thru 0x05ad88 (0xad89 bytes) 000204[1] 1753364871.385868 read 0x4c3000 thru 0x4c4278 (0x1279 bytes) 000205[0] 1753364871.397112 mapread 0x1ce000 thru 0x1cf2e1 (0x12e2 bytes) 000208[0] 1753364871.403030 trunc from 0x51b2a1 to 0x8b8665 (0x39d3c5 bytes) 000209[0] 1753364871.419692 write 0x672000 thru 0x678509 (0x650a bytes) 000211[1] 1753364871.429077 mapwrite 0x900000 thru 0x90ab9a (0xab9b bytes) 000213[1] 1753364871.474979 read 0x268000 thru 0x270aea (0x8aeb bytes) 000214[0] 1753364871.479548 mapwrite 0x7e3000 thru 0x7f2704 (0xf705 bytes) 000215[1] 1753364871.528911 mapwrite 0x259000 thru 0x268144 (0xf145 bytes) 000217[0] 1753364871.594230 trunc from 0x90ab9b to 0x7e8993 (0x122207 bytes) 000218[0] 1753364871.606113 read 0x6e3000 thru 0x6f05bc (0xd5bd bytes) 000219[1] 1753364871.611274 mapread 0x2a5000 thru 0x2aedb3 (0x9db4 bytes) 000222[0] 1753364871.630120 write 0x5b3000 thru 0x5bb0c9 (0x80ca bytes) 000223[1] 1753364871.641119 trunc from 0x7e8994 to 0x1a225f (0x646734 bytes) 000225[0] 1753364871.659114 trunc from 0x1a2260 to 0x50722a (0x364fcb bytes) 000226[0] 1753364871.669570 read 0x3f1000 thru 0x3f21ae (0x11af bytes) 000228[0] 1753364871.681122 trunc from 0x50722b to 0x009f76 (0x4fd2b4 bytes) 000233[0] 1753364871.710285 mapwrite 0x26c000 thru 0x26cf36 (0xf37 bytes) 000234[1] 1753364871.745353 mapwrite 0x9e1000 thru 0x9eb87e (0xa87f bytes) 000235[0] 1753364871.790479 mapwrite 0x55a000 thru 0x565e97 (0xbe98 bytes) 000241[0] 1753364871.835262 trunc from 0x9eb87f to 0x72a71a (0x2c1164 bytes) 000242[0] 1753364871.846382 mapwrite 0x088000 thru 0x092ac3 (0xaac4 bytes) 000245[0] 1753364871.873057 mapread 0x16b000 thru 0x16e0e2 (0x30e3 bytes) 000249[0] 1753364871.882598 mapwrite 0x8eb000 thru 0x8f1fc5 (0x6fc6 bytes) 000251[1] 1753364871.924699 read 0x535000 thru 0x536bb3 (0x1bb4 bytes) 000252[1] 1753364871.931218 mapread 0x413000 thru 0x41516f (0x2170 bytes) 000254[1] 1753364871.940514 trunc from 0x8f1fc6 to 0x37ab4e (0x577477 bytes) 000259[1] 1753364871.957299 mapwrite 0x0a2000 thru 0x0a5187 (0x3188 bytes) 000260[0] 1753364871.986685 trunc from 0x37ab4f to 0x0cb718 (0x2af436 bytes) 000262[1] 1753364871.999967 read 0x06a000 thru 0x06c502 (0x2503 bytes) 000264[0] 1753364872.016166 trunc from 0x0cb719 to 0x2cc473 (0x200d5b bytes) 000266[1] 1753364872.036684 mapwrite 0x28f000 thru 0x28ffae (0xfaf bytes) 000268[0] 1753364872.079612 mapwrite 0x26b000 thru 0x272be7 (0x7be8 bytes) 000269[1] 1753364872.122012 write 0x8bf000 thru 0x8c967d (0xa67e bytes) 000271[0] 1753364872.134524 trunc from 0x8c967e to 0x3dacf5 (0x4ee988 bytes) 000272[0] 1753364872.153273 mapread 0x3d2000 thru 0x3d521d (0x321e bytes) 000273[0] 1753364872.163449 read 0x22e000 thru 0x2385fa (0xa5fb bytes) 000277[1] 1753364872.201397 read 0x1b1000 thru 0x1b43ed (0x33ee bytes) 000279[1] 1753364872.242148 mapwrite 0x8cb000 thru 0x8cd8ab (0x28ac bytes) 000280[1] 1753364872.301339 read 0x394000 thru 0x3a24da (0xe4db bytes) 000283[1] 1753364872.312030 read 0x283000 thru 0x289273 (0x6274 bytes) 000285[0] 1753364872.314609 mapwrite 0x53d000 thru 0x544788 (0x7789 bytes) 000286[0] 1753364872.371093 read 0x866000 thru 0x86a83d (0x483e bytes) 000287[1] 1753364872.374530 mapread 0x5c2000 thru 0x5d1a76 (0xfa77 bytes) 000288[1] 1753364872.394690 mapwrite 0x417000 thru 0x4255d4 (0xe5d5 bytes) 000290[0] 1753364872.434232 mapread 0x7c6000 thru 0x7ca8fb (0x48fc bytes) 000292[0] 1753364872.444477 write 0x864000 thru 0x867162 (0x3163 bytes) 000296[0] 1753364872.462526 trunc from 0x8cd8ac to 0x285a1f (0x647e8c bytes) 000298[1] 1753364872.473623 write 0x628000 thru 0x636072 (0xe073 bytes) 000301[1] 1753364872.482408 write 0x638000 thru 0x641e4b (0x9e4c bytes) 000302[0] 1753364872.487279 mapwrite 0x088000 thru 0x08df78 (0x5f79 bytes) 000305[1] 1753364872.530423 mapread 0x2ec000 thru 0x2f33fe (0x73ff bytes) 000306[1] 1753364872.541711 mapread 0x414000 thru 0x4164b6 (0x24b7 bytes) 000311[1] 1753364872.550337 mapread 0x40f000 thru 0x412154 (0x3155 bytes) 000312[1] 1753364872.550826 read 0x06b000 thru 0x0707c6 (0x57c7 bytes) 000313[1] 1753364872.553891 mapwrite 0x941000 thru 0x948527 (0x7528 bytes) 000314[0] 1753364872.586459 mapwrite 0x478000 thru 0x47a276 (0x2277 bytes) truncating to largest ever: 0x9e7a3d 000315[0] 1753364872.616950 trunc from 0x948528 to 0x9e7a3c (0x9f515 bytes) 000316[0] 1753364872.622188 mapwrite 0x32f000 thru 0x333cc0 (0x4cc1 bytes) 000319[1] 1753364872.655713 read 0x18c000 thru 0x198e69 (0xce6a bytes) 000322[0] 1753364872.664866 mapread 0x00e000 thru 0x0152e9 (0x72ea bytes) 000323[1] 1753364872.671329 trunc from 0x9e7a3d to 0x161ce0 (0x885d5c bytes) 000324[0] 1753364872.682374 mapwrite 0x370000 thru 0x371d66 (0x1d67 bytes) 000329[1] 1753364872.732061 mapwrite 0x4fc000 thru 0x509f31 (0xdf32 bytes) 000330[0] 1753364872.772115 write 0x738000 thru 0x7404c0 (0x84c1 bytes) 000331[1] 1753364872.781285 mapread 0x222000 thru 0x22a61c (0x861d bytes) 000332[1] 1753364872.795299 write 0x9f5000 thru 0x9f7f34 (0x2f35 bytes) 000333[0] 1753364872.805617 write 0x66a000 thru 0x66c610 (0x2611 bytes) 000335[1] 1753364872.822061 trunc from 0x9f7f35 to 0x95768d (0xa08a7 bytes) 000336[0] 1753364872.833349 read 0x182000 thru 0x18b24e (0x924f bytes) 000338[0] 1753364872.841753 trunc from 0x95768e to 0x36710f (0x5f057e bytes) 000339[1] 1753364872.853117 write 0x337000 thru 0x3446eb (0xd6ec bytes) 000342[0] 1753364872.875702 trunc from 0x367110 to 0x8467b5 (0x4df6a6 bytes) 000343[1] 1753364872.897271 mapread 0x568000 thru 0x576440 (0xe441 bytes) 000348[0] 1753364872.915935 mapwrite 0x4f8000 thru 0x4fe8d6 (0x68d7 bytes) 000350[1] 1753364872.947229 write 0x9d3000 thru 0x9ddeec (0xaeed bytes) 000351[0] 1753364872.955075 mapwrite 0x073000 thru 0x08218c (0xf18d bytes) 000352[1] 1753364872.991423 mapread 0x44b000 thru 0x457e8f (0xce90 bytes) 000353[0] 1753364873.006859 mapwrite 0x184000 thru 0x18dd3a (0x9d3b bytes) 000354[0] 1753364873.042186 mapwrite 0x4ca000 thru 0x4d3288 (0x9289 bytes) 000355[0] 1753364873.067479 trunc from 0x9ddeed to 0x83f646 (0x19e8a6 bytes) 000357[1] 1753364873.082848 read 0x672000 thru 0x679900 (0x7901 bytes) 000358[1] 1753364873.093418 mapwrite 0x29d000 thru 0x2aa4a8 (0xd4a9 bytes) 000360[0] 1753364873.143278 trunc from 0x83f647 to 0x6adb39 (0x191b0d bytes) 000361[0] 1753364873.156180 trunc from 0x6adb3a to 0x5e0a84 (0xcd0b5 bytes) 000363[1] 1753364873.168188 read 0x02c000 thru 0x02f56d (0x356e bytes) 000364[1] 1753364873.173180 write 0x87d000 thru 0x87d5e4 (0x5e5 bytes) 000368[0] 1753364873.192984 read 0x3bf000 thru 0x3ce1ec (0xf1ed bytes) 000372[0] 1753364873.203931 trunc from 0x87d5e5 to 0x8d2dad (0x557c9 bytes) 000374[0] 1753364873.223539 mapread 0x7d0000 thru 0x7d562b (0x562c bytes) 000381[0] 1753364873.251819 mapread 0x88b000 thru 0x88e3e8 (0x33e9 bytes) 000382[0] 1753364873.258834 mapread 0x2dd000 thru 0x2e5df1 (0x8df2 bytes) 000383[0] 1753364873.268731 trunc from 0x8d2dae to 0x915e4c (0x4309f bytes) 000384[1] 1753364873.274374 write 0x449000 thru 0x44f2b7 (0x62b8 bytes) 000386[0] 1753364873.281063 write 0x9a1000 thru 0x9a30d8 (0x20d9 bytes) 000389[0] 1753364873.294409 mapwrite 0x723000 thru 0x72a102 (0x7103 bytes) 000390[0] 1753364873.317162 mapread 0x353000 thru 0x356e88 (0x3e89 bytes) 000392[0] 1753364873.324764 write 0x512000 thru 0x517268 (0x5269 bytes) 000393[0] 1753364873.331223 mapread 0x1a3000 thru 0x1ae84c (0xb84d bytes) 000397[1] 1753364873.349526 read 0x701000 thru 0x704960 (0x3961 bytes) 000399[1] 1753364873.357753 mapread 0x48a000 thru 0x48ec5f (0x4c60 bytes) 000400[1] 1753364873.365777 mapwrite 0x685000 thru 0x68dcd9 (0x8cda bytes) 000401[1] 1753364873.398134 mapwrite 0x810000 thru 0x81711e (0x711f bytes) 000402[1] 1753364873.429082 write 0x82d000 thru 0x83a7b6 (0xd7b7 bytes) 000405[0] 1753364873.432506 mapread 0x815000 thru 0x818d79 (0x3d7a bytes) 000407[1] 1753364873.449511 trunc from 0x9a30d9 to 0x184cf2 (0x81e3e6 bytes) 000408[1] 1753364873.465529 mapread 0x0d2000 thru 0x0d3f7b (0x1f7c bytes) 000410[1] 1753364873.480384 read 0x11d000 thru 0x129c8b (0xcc8c bytes) 000411[0] 1753364873.501139 trunc from 0x184cf3 to 0x6e936d (0x56467b bytes) 000412[1] 1753364873.519558 mapread 0x1fc000 thru 0x20ae5c (0xee5d bytes) 000414[1] 1753364873.555789 trunc from 0x6e936e to 0x48ea8d (0x25a8e0 bytes) 000415[0] 1753364873.569713 read 0x26d000 thru 0x26d5d6 (0x5d7 bytes) 000416[1] 1753364873.576968 read 0x397000 thru 0x3a3122 (0xc123 bytes) 000418[1] 1753364873.582376 mapwrite 0x0d2000 thru 0x0d9d6d (0x7d6e bytes) 000420[1] 1753364873.614418 trunc from 0x48ea8e to 0x692094 (0x203607 bytes) 000421[0] 1753364873.627292 read 0x4c1000 thru 0x4d02bc (0xf2bd bytes) 000422[0] 1753364873.634770 mapwrite 0x91c000 thru 0x926aa9 (0xaaaa bytes) 000424[1] 1753364873.689931 read 0x35b000 thru 0x35f0e9 (0x40ea bytes) 000426[0] 1753364873.700005 mapread 0x70b000 thru 0x718dd8 (0xddd9 bytes) 000430[0] 1753364873.712527 mapread 0x361000 thru 0x368c0a (0x7c0b bytes) 000431[1] 1753364873.719473 read 0x853000 thru 0x85b63e (0x863f bytes) 000433[1] 1753364873.723241 write 0x532000 thru 0x5385e2 (0x65e3 bytes) 000435[1] 1753364873.742657 trunc from 0x926aaa to 0x5a0ec0 (0x385be9 bytes) 000436[1] 1753364873.755993 write 0x6c4000 thru 0x6d3673 (0xf674 bytes) 000437[0] 1753364873.765071 write 0x6d9000 thru 0x6e07c9 (0x77ca bytes) 000438[1] 1753364873.778905 trunc from 0x6e07ca to 0x72f79f (0x4efd6 bytes) 000439[1] 1753364873.800657 read 0x10d000 thru 0x11a79d (0xd79e bytes) 000440[0] 1753364873.806653 write 0x5ba000 thru 0x5c87c1 (0xe7c2 bytes) 000444[0] 1753364873.819999 mapread 0x6e4000 thru 0x6e6d81 (0x2d82 bytes) 000446[0] 1753364873.833483 mapwrite 0x09a000 thru 0x0a0a5e (0x6a5f bytes) 000447[0] 1753364873.865698 write 0x452000 thru 0x45d171 (0xb172 bytes) 000449[1] 1753364873.879724 trunc from 0x72f7a0 to 0x1efa44 (0x53fd5b bytes) 000452[0] 1753364873.902136 write 0x375000 thru 0x3848c5 (0xf8c6 bytes) 000454[1] 1753364873.916055 mapread 0x14f000 thru 0x158ac5 (0x9ac6 bytes) 000456[0] 1753364873.933375 mapwrite 0x41d000 thru 0x41d8a6 (0x8a7 bytes) 000457[0] 1753364873.966873 mapread 0x2ab000 thru 0x2b27d4 (0x77d5 bytes) 000465[0] 1753364874.014492 mapwrite 0x942000 thru 0x943517 (0x1518 bytes) 000466[0] 1753364874.051429 mapwrite 0x3fa000 thru 0x4066ef (0xc6f0 bytes) 000468[0] 1753364874.091723 trunc from 0x943518 to 0x90094c (0x42bcb bytes) 000471[1] 1753364874.115306 mapread 0x767000 thru 0x767c9f (0xca0 bytes) 000477[1] 1753364874.127013 mapwrite 0x654000 thru 0x662e58 (0xee59 bytes) 000478[1] 1753364874.171421 trunc from 0x90094d to 0x8d5b62 (0x2adea bytes) 000480[0] 1753364874.186773 read 0x392000 thru 0x39b231 (0x9232 bytes) 000482[1] 1753364874.194479 mapwrite 0x92a000 thru 0x931815 (0x7816 bytes) 000483[1] 1753364874.242124 trunc from 0x931816 to 0x2aa8f9 (0x686f1c bytes) 000486[1] 1753364874.251867 trunc from 0x2aa8fa to 0x370919 (0xc6020 bytes) 000490[0] 1753364874.275369 mapwrite 0x624000 thru 0x6321c8 (0xe1c9 bytes) 000491[1] 1753364874.322850 read 0x027000 thru 0x0286e0 (0x16e1 bytes) 000492[0] 1753364874.332086 read 0x0ef000 thru 0x0fc2f9 (0xd2fa bytes) 000493[0] 1753364874.335957 mapwrite 0x1ab000 thru 0x1b1514 (0x6515 bytes) 000495[0] 1753364874.367499 mapwrite 0x64b000 thru 0x659c81 (0xec82 bytes) 000496[0] 1753364874.402705 trunc from 0x659c82 to 0x0535d4 (0x6066ad bytes) 000497[1] 1753364874.413572 read 0x01b000 thru 0x027182 (0xc183 bytes) 000498[1] 1753364874.425735 mapread 0x03a000 thru 0x045a49 (0xba4a bytes) 000500[0] 1753364874.426670 read 0x03d000 thru 0x044ccd (0x7cce bytes) All operations completed A-OK! PASS 16a (16s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:47:56 (1753364876) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.211802 s, 39.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.186124 s, 45.1 MB/s Chance of close/open is 1 in 50 Seed set to 4877 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000006[0] 1753364878.296219 mapwrite 0x2b378b thru 0x2b401f (0x895 bytes) 000007[1] 1753364878.347273 write 0x220b8e thru 0x221e04 (0x1277 bytes) 000008[0] 1753364878.363253 read 0x0b345b thru 0x0b4d90 (0x1936 bytes) 000009[0] 1753364878.383052 mapwrite 0x990df0 thru 0x991a09 (0xc1a bytes) 000013[1] 1753364878.466461 read 0x7aaacf thru 0x7ab5c1 (0xaf3 bytes) 000019[0] 1753364878.515203 mapwrite 0x39bce9 thru 0x39db1c (0x1e34 bytes) 000020[0] 1753364878.549160 mapread 0x15e621 thru 0x15ff65 (0x1945 bytes) 000021[1] 1753364878.555023 read 0x6e60d5 thru 0x6e6f8e (0xeba bytes) truncating to largest ever: 0x49f191 000023[1] 1753364878.559030 trunc from 0x991a0a to 0x49f190 (0x4f2879 bytes) truncating to largest ever: 0x651dce 000024[0] 1753364878.585600 trunc from 0x49f191 to 0x651dcd (0x1b2c3d bytes) 000026[1] 1753364878.622655 mapread 0x368d25 thru 0x369360 (0x63c bytes) truncating to largest ever: 0x843743 000027[0] 1753364878.640739 trunc from 0x651dce to 0x843742 (0x1f1975 bytes) 000028[1] 1753364878.656209 mapread 0x300b64 thru 0x302108 (0x15a5 bytes) 000029[1] 1753364878.678371 mapwrite 0x52f220 thru 0x53090f (0x16f0 bytes) 000030[1] 1753364878.737463 mapread 0x471eb3 thru 0x472c2d (0xd7b bytes) 000031[1] 1753364878.754627 write 0x539b68 thru 0x53b70b (0x1ba4 bytes) 000033[0] 1753364878.775286 mapwrite 0x2a326f thru 0x2a37b3 (0x545 bytes) 000038[1] 1753364878.849630 trunc from 0x843743 to 0x566cf7 (0x2dca4b bytes) 000039[1] 1753364878.863070 mapwrite 0x0d4c21 thru 0x0d6086 (0x1466 bytes) 000040[1] 1753364878.925165 mapwrite 0x944cf6 thru 0x945552 (0x85d bytes) 000041[1] 1753364878.982704 trunc from 0x945553 to 0x069957 (0x8dbbfb bytes) 000044[0] 1753364879.021176 mapread 0x025013 thru 0x02611e (0x110c bytes) 000045[1] 1753364879.038469 read 0x00b5a1 thru 0x00c1b0 (0xc10 bytes) 000046[1] 1753364879.057997 mapwrite 0x5699a0 thru 0x56aaeb (0x114c bytes) 000047[1] 1753364879.141146 trunc from 0x56aaec to 0x78e1d7 (0x2236ec bytes) 000055[1] 1753364879.216336 mapread 0x492721 thru 0x49350e (0xdee bytes) 000056[0] 1753364879.225269 write 0x6d4227 thru 0x6d52e6 (0x10c0 bytes) 000061[1] 1753364879.268646 mapread 0x573ca7 thru 0x5749cd (0xd27 bytes) 000063[0] 1753364879.301163 read 0x2fb838 thru 0x2fc371 (0xb3a bytes) 000065[1] 1753364879.317297 mapwrite 0x157f31 thru 0x158c49 (0xd19 bytes) 000066[1] 1753364879.359945 trunc from 0x78e1d8 to 0x3e0e88 (0x3ad34f bytes) 000068[1] 1753364879.399754 mapread 0x1cc6ca thru 0x1ce07e (0x19b5 bytes) 000069[1] 1753364879.415968 mapwrite 0x3de0a6 thru 0x3de964 (0x8bf bytes) 000070[0] 1753364879.449759 trunc from 0x3e0e89 to 0x386768 (0x5a720 bytes) 000071[0] 1753364879.469528 mapread 0x078ced thru 0x07a047 (0x135b bytes) 000072[1] 1753364879.478770 mapread 0x0eb0ec thru 0x0ebd1e (0xc33 bytes) 000075[1] 1753364879.501836 mapread 0x2555a8 thru 0x256f41 (0x199a bytes) 000077[1] 1753364879.512812 mapread 0x25330d thru 0x2535cc (0x2c0 bytes) 000079[0] 1753364879.520535 read 0x1f5cef thru 0x1f7594 (0x18a6 bytes) 000080[1] 1753364879.553222 write 0x0a5304 thru 0x0a541e (0x11b bytes) 000082[0] 1753364879.587613 trunc from 0x386769 to 0x356aae (0x2fcba bytes) 000083[1] 1753364879.620941 read 0x14ff7b thru 0x150962 (0x9e8 bytes) 000084[1] 1753364879.683508 write 0x77677c thru 0x776bc8 (0x44d bytes) 000085[0] 1753364879.694938 write 0x8fd8c3 thru 0x8fe851 (0xf8f bytes) 000086[0] 1753364879.710953 trunc from 0x8fe852 to 0x0e6ccf (0x817b82 bytes) 000087[1] 1753364879.726397 mapread 0x07cc3f thru 0x07d058 (0x41a bytes) 000090[0] 1753364879.752456 mapread 0x083351 thru 0x085100 (0x1db0 bytes) 000094[0] 1753364879.778476 trunc from 0x0e6cd0 to 0x3ec68a (0x3059bb bytes) 000096[1] 1753364879.802204 mapread 0x2456cd thru 0x247099 (0x19cd bytes) 000097[1] 1753364879.832366 write 0x976fde thru 0x9770ee (0x111 bytes) 000098[0] 1753364879.851296 read 0x145e25 thru 0x146e03 (0xfdf bytes) 000099[1] 1753364879.899991 read 0x74da91 thru 0x74e235 (0x7a5 bytes) 000100[0] 1753364879.915988 trunc from 0x9770ef to 0x2e703b (0x6900b3 bytes) 000101[1] 1753364879.950619 mapwrite 0x58cc13 thru 0x58ce7d (0x26b bytes) 000102[0] 1753364880.006396 mapread 0x397056 thru 0x3971f7 (0x1a2 bytes) 000103[1] 1753364880.032971 read 0x19bf15 thru 0x19d32d (0x1419 bytes) 000105[1] 1753364880.049919 read 0x2d7f72 thru 0x2d9365 (0x13f4 bytes) 000106[1] 1753364880.054866 write 0x612607 thru 0x612808 (0x202 bytes) 000109[0] 1753364880.067482 write 0x49406e thru 0x494957 (0x8ea bytes) 000113[1] 1753364880.133346 write 0x5fb670 thru 0x5fcf4f (0x18e0 bytes) 000114[0] 1753364880.163364 read 0x190db9 thru 0x190fde (0x226 bytes) 000115[1] 1753364880.182400 mapwrite 0x0c8902 thru 0x0c91af (0x8ae bytes) 000116[1] 1753364880.231558 mapwrite 0x4e79e8 thru 0x4e8f3e (0x1557 bytes) 000117[0] 1753364880.277037 write 0x943864 thru 0x9454e9 (0x1c86 bytes) 000120[0] 1753364880.331146 mapread 0x82d425 thru 0x82d785 (0x361 bytes) 000121[1] 1753364880.354724 trunc from 0x9454ea to 0x3e74c2 (0x55e027 bytes) 000122[0] 1753364880.377731 trunc from 0x3e74c3 to 0x363030 (0x84492 bytes) 000123[1] 1753364880.400086 trunc from 0x363031 to 0x626f84 (0x2c3f54 bytes) 000124[1] 1753364880.438429 read 0x4c77d3 thru 0x4c7e22 (0x650 bytes) 000125[0] 1753364880.468407 mapread 0x209082 thru 0x209783 (0x702 bytes) 000126[1] 1753364880.509228 mapwrite 0x42c26d thru 0x42cab9 (0x84d bytes) 000128[1] 1753364880.563928 write 0x0a79c4 thru 0x0a8a67 (0x10a4 bytes) 000129[0] 1753364880.596009 mapwrite 0x4b8d28 thru 0x4b9738 (0xa11 bytes) 000130[0] 1753364880.661604 read 0x473304 thru 0x474a63 (0x1760 bytes) 000131[1] 1753364880.680528 trunc from 0x626f85 to 0x2f786f (0x32f715 bytes) 000134[0] 1753364880.753551 mapwrite 0x55e488 thru 0x5603bf (0x1f38 bytes) 000135[0] 1753364880.820095 write 0x746494 thru 0x7467ee (0x35b bytes) truncating to largest ever: 0x8b63db 000138[1] 1753364880.851326 trunc from 0x7467ef to 0x8b63da (0x16fbec bytes) 000139[1] 1753364880.880295 trunc from 0x8b63db to 0x45e4cc (0x457f0e bytes) 000143[1] 1753364880.905346 mapwrite 0x041c35 thru 0x04392a (0x1cf6 bytes) 000144[0] 1753364880.940168 read 0x164515 thru 0x166093 (0x1b7f bytes) 000148[0] 1753364880.953547 read 0x3ff934 thru 0x400a33 (0x1100 bytes) 000150[1] 1753364880.976430 mapwrite 0x36b2e6 thru 0x36b74c (0x467 bytes) 000158[0] 1753364881.062567 mapread 0x0ead5c thru 0x0ec140 (0x13e5 bytes) 000160[0] 1753364881.101441 trunc from 0x45e4cd to 0x5636d6 (0x10520a bytes) 000161[1] 1753364881.128679 mapwrite 0x183f44 thru 0x185b24 (0x1be1 bytes) 000163[0] 1753364881.194939 mapread 0x0942f4 thru 0x094dc5 (0xad2 bytes) 000166[0] 1753364881.220196 mapread 0x342672 thru 0x3427ed (0x17c bytes) 000167[0] 1753364881.227675 mapwrite 0x121634 thru 0x122d70 (0x173d bytes) 000168[1] 1753364881.272265 write 0x3d3e6f thru 0x3d5971 (0x1b03 bytes) 000169[1] 1753364881.293310 mapwrite 0x4b09ab thru 0x4b2365 (0x19bb bytes) 000171[0] 1753364881.345025 read 0x49e860 thru 0x4a011d (0x18be bytes) 000176[0] 1753364881.373579 write 0x615d56 thru 0x615ef0 (0x19b bytes) truncating to largest ever: 0x950c49 000177[1] 1753364881.389334 trunc from 0x615ef1 to 0x950c48 (0x33ad58 bytes) 000180[0] 1753364881.412882 write 0x33fe4e thru 0x3402ea (0x49d bytes) 000181[0] 1753364881.447428 trunc from 0x950c49 to 0x613618 (0x33d630 bytes) 000182[1] 1753364881.479630 write 0x256760 thru 0x256943 (0x1e4 bytes) 000184[0] 1753364881.516722 mapwrite 0x6a3bed thru 0x6a5367 (0x177b bytes) 000188[1] 1753364881.631476 read 0x1002af thru 0x100e24 (0xb76 bytes) 000190[0] 1753364881.646104 read 0x0df70a thru 0x0e09c4 (0x12bb bytes) 000192[0] 1753364881.650645 mapread 0x3cd64d thru 0x3cea95 (0x1449 bytes) 000194[0] 1753364881.667957 mapwrite 0x8d389b thru 0x8d3b58 (0x2be bytes) 000200[0] 1753364881.765624 read 0x0eec03 thru 0x0f05a5 (0x19a3 bytes) 000201[1] 1753364881.768657 read 0x3df4dc thru 0x3df603 (0x128 bytes) 000202[0] 1753364881.784692 trunc from 0x8d3b59 to 0x7217f6 (0x1b2362 bytes) 000204[1] 1753364881.801361 read 0x19e90a thru 0x19ee3b (0x532 bytes) 000205[0] 1753364881.823896 read 0x628bae thru 0x62a710 (0x1b63 bytes) 000206[0] 1753364881.829171 mapread 0x1d0729 thru 0x1d145f (0xd37 bytes) 000210[0] 1753364881.844762 mapread 0x4a4965 thru 0x4a591f (0xfbb bytes) 000211[0] 1753364881.859418 trunc from 0x7217f7 to 0x06394f (0x6bdea7 bytes) 000213[0] 1753364881.889471 read 0x01aa4e thru 0x01c09c (0x164f bytes) 000215[0] 1753364881.908112 mapwrite 0x93a16e thru 0x93a9e3 (0x876 bytes) 000216[0] 1753364881.998987 trunc from 0x93a9e4 to 0x830959 (0x10a08a bytes) 000218[1] 1753364882.026905 mapread 0x0014fe thru 0x002f7e (0x1a81 bytes) 000221[0] 1753364882.060073 mapread 0x159abd thru 0x15a782 (0xcc6 bytes) 000222[1] 1753364882.066213 write 0x6b17db thru 0x6b33a6 (0x1bcc bytes) 000224[1] 1753364882.084894 mapread 0x012983 thru 0x01394b (0xfc9 bytes) 000226[1] 1753364882.097113 mapwrite 0x81f9e8 thru 0x8218b6 (0x1ecf bytes) 000230[1] 1753364882.156400 read 0x816645 thru 0x8168b7 (0x273 bytes) 000231[1] 1753364882.160280 read 0x0ab540 thru 0x0ac164 (0xc25 bytes) 000233[1] 1753364882.179967 read 0x3db2f3 thru 0x3dc68e (0x139c bytes) 000234[1] 1753364882.183747 read 0x0a2bbd thru 0x0a3063 (0x4a7 bytes) 000236[0] 1753364882.199376 mapwrite 0x02b59d thru 0x02cd0b (0x176f bytes) 000238[0] 1753364882.274633 trunc from 0x83095a to 0x74df2c (0xe2a2d bytes) 000241[0] 1753364882.324183 read 0x27b8e0 thru 0x27d59c (0x1cbd bytes) 000244[0] 1753364882.336717 write 0x607192 thru 0x6089e4 (0x1853 bytes) 000247[0] 1753364882.364007 mapread 0x4f5171 thru 0x4f5202 (0x92 bytes) 000248[0] 1753364882.385604 write 0x656d44 thru 0x6570f9 (0x3b6 bytes) 000249[1] 1753364882.403863 read 0x5af735 thru 0x5b1322 (0x1bee bytes) 000252[1] 1753364882.445067 trunc from 0x74df2d to 0x72e562 (0x1f9ca bytes) 000253[0] 1753364882.477990 write 0x8d7256 thru 0x8d7f60 (0xd0b bytes) 000255[0] 1753364882.503581 read 0x18b830 thru 0x18bc8f (0x460 bytes) 000256[1] 1753364882.517713 mapwrite 0x2e959a thru 0x2e9c2e (0x695 bytes) 000258[0] 1753364882.580454 write 0x7caa1f thru 0x7cbdc6 (0x13a8 bytes) 000263[0] 1753364882.642391 mapread 0x5a184f thru 0x5a185a (0xc bytes) 000264[0] 1753364882.662206 mapwrite 0x819ce6 thru 0x81a7ef (0xb0a bytes) 000266[1] 1753364882.721075 trunc from 0x8d7f61 to 0x82c3e4 (0xabb7c bytes) 000267[1] 1753364882.748751 write 0x6320f4 thru 0x632116 (0x23 bytes) 000268[0] 1753364882.751417 mapwrite 0x37f2d2 thru 0x3809a6 (0x16d5 bytes) 000270[1] 1753364882.812750 mapread 0x2085a7 thru 0x20917b (0xbd5 bytes) 000274[1] 1753364882.856564 trunc from 0x82c3e5 to 0x44406e (0x3e8376 bytes) 000275[1] 1753364882.886094 mapwrite 0x38a45a thru 0x38c2cd (0x1e74 bytes) 000276[0] 1753364882.935232 mapwrite 0x83ae9b thru 0x83c44c (0x15b2 bytes) 000277[1] 1753364883.004231 read 0x205b41 thru 0x207485 (0x1945 bytes) 000278[0] 1753364883.024747 mapwrite 0x07209d thru 0x072b6d (0xad1 bytes) 000280[0] 1753364883.089203 mapwrite 0x468398 thru 0x468cdc (0x945 bytes) 000281[1] 1753364883.154865 mapread 0x7a2d9c thru 0x7a3bf9 (0xe5e bytes) 000286[0] 1753364883.222703 mapread 0x11495d thru 0x114a29 (0xcd bytes) 000289[1] 1753364883.228453 mapwrite 0x462318 thru 0x46376d (0x1456 bytes) 000290[1] 1753364883.290064 mapread 0x1c8e9c thru 0x1c96ca (0x82f bytes) 000291[0] 1753364883.312038 mapwrite 0x3763a5 thru 0x376cd2 (0x92e bytes) 000293[1] 1753364883.368687 read 0x8026b2 thru 0x8031e1 (0xb30 bytes) 000295[1] 1753364883.385733 mapread 0x79616e thru 0x797bf4 (0x1a87 bytes) 000296[1] 1753364883.403581 trunc from 0x83c44d to 0x29b799 (0x5a0cb3 bytes) 000297[1] 1753364883.427966 mapread 0x25d991 thru 0x25f093 (0x1703 bytes) 000299[0] 1753364883.479149 mapread 0x23e98c thru 0x23f703 (0xd78 bytes) 000302[0] 1753364883.567336 mapread 0x04b551 thru 0x04bf55 (0xa05 bytes) 000303[1] 1753364883.585705 trunc from 0x29b79a to 0x105cbd (0x195adc bytes) 000305[1] 1753364883.648434 write 0x7aa7d1 thru 0x7ab156 (0x986 bytes) 000306[0] 1753364883.668211 mapwrite 0x20ef61 thru 0x210412 (0x14b2 bytes) 000311[1] 1753364883.856053 mapread 0x76cb89 thru 0x76d8e3 (0xd5b bytes) 000313[1] 1753364883.927639 mapread 0x0d19ce thru 0x0d308a (0x16bd bytes) 000318[0] 1753364883.979268 mapread 0x0cfda3 thru 0x0d17f8 (0x1a56 bytes) 000319[1] 1753364883.999558 mapwrite 0x8dd296 thru 0x8dd554 (0x2bf bytes) 000320[1] 1753364884.069603 mapread 0x2404ac thru 0x241c57 (0x17ac bytes) 000322[0] 1753364884.108476 trunc from 0x8dd555 to 0x52b446 (0x3b210e bytes) 000323[1] 1753364884.157903 read 0x19035f thru 0x190fec (0xc8e bytes) 000325[0] 1753364884.219058 write 0x5dd45a thru 0x5df0f8 (0x1c9f bytes) 000326[1] 1753364884.236370 read 0x3718eb thru 0x373879 (0x1f8f bytes) 000327[0] 1753364884.253152 write 0x6880b4 thru 0x6892f2 (0x123f bytes) 000332[0] 1753364884.286609 write 0x6d7772 thru 0x6d88e6 (0x1175 bytes) 000333[0] 1753364884.307895 read 0x038199 thru 0x038f1f (0xd87 bytes) 000336[0] 1753364884.351390 read 0x4503cb thru 0x45209a (0x1cd0 bytes) 000337[0] 1753364884.358143 mapwrite 0x304e53 thru 0x30599e (0xb4c bytes) 000342[0] 1753364884.475085 write 0x42ec05 thru 0x42f15d (0x559 bytes) 000345[1] 1753364884.543721 read 0x0ec56c thru 0x0eca5a (0x4ef bytes) 000347[0] 1753364884.588331 mapwrite 0x237c68 thru 0x238027 (0x3c0 bytes) 000349[1] 1753364884.655710 mapread 0x3263ec thru 0x3266d3 (0x2e8 bytes) 000350[1] 1753364884.683078 mapwrite 0x747b95 thru 0x747fc2 (0x42e bytes) 000351[0] 1753364884.744992 read 0x382b8e thru 0x38478f (0x1c02 bytes) 000352[0] 1753364884.768181 write 0x13b444 thru 0x13c8a4 (0x1461 bytes) 000356[0] 1753364884.839909 write 0x9e71fc thru 0x9e7272 (0x77 bytes) 000357[1] 1753364884.856594 write 0x11ee6c thru 0x11f15b (0x2f0 bytes) 000359[1] 1753364884.915187 read 0x586b9e thru 0x588aac (0x1f0f bytes) 000361[1] 1753364884.930983 write 0x58afea thru 0x58c929 (0x1940 bytes) truncating to largest ever: 0x9724e1 000365[0] 1753364884.956846 trunc from 0x9e7273 to 0x9724e0 (0x74d92 bytes) 000366[1] 1753364884.980157 mapread 0x47c5bf thru 0x47e278 (0x1cba bytes) 000367[1] 1753364885.006885 write 0x610f97 thru 0x611287 (0x2f1 bytes) skipping zero size read 000371[0] 1753364885.057548 mapread 0x4f0a33 thru 0x4f221f (0x17ed bytes) 000372[0] 1753364885.082933 write 0x49b8bd thru 0x49d236 (0x197a bytes) 000376[1] 1753364885.126655 trunc from 0x9724e1 to 0x50ac24 (0x4678bc bytes) 000377[1] 1753364885.161787 mapwrite 0x1f42b8 thru 0x1f5417 (0x1160 bytes) 000379[0] 1753364885.224476 read 0x049023 thru 0x049b11 (0xaef bytes) 000382[0] 1753364885.264256 mapread 0x129639 thru 0x12a6ee (0x10b6 bytes) 000385[0] 1753364885.289713 write 0x7f8859 thru 0x7f8fa9 (0x751 bytes) 000387[1] 1753364885.307163 write 0x7ac20f thru 0x7adfd9 (0x1dcb bytes) 000388[1] 1753364885.341891 write 0x54b315 thru 0x54d11d (0x1e09 bytes) 000389[1] 1753364885.351678 mapwrite 0x3a3130 thru 0x3a320c (0xdd bytes) 000390[0] 1753364885.401518 write 0x0310b0 thru 0x03152b (0x47c bytes) 000393[1] 1753364885.443264 trunc from 0x7f8faa to 0x4445d6 (0x3b49d3 bytes) 000396[1] 1753364885.486600 mapread 0x003480 thru 0x0040ab (0xc2c bytes) 000397[1] 1753364885.494688 trunc from 0x4445d7 to 0x64eb36 (0x20a560 bytes) 000400[0] 1753364885.524742 trunc from 0x64eb37 to 0x27f576 (0x3cf5c0 bytes) 000404[0] 1753364885.592500 write 0x90a0b8 thru 0x90b228 (0x1171 bytes) 000406[0] 1753364885.609554 trunc from 0x90b229 to 0x6a992f (0x2618f9 bytes) 000412[0] 1753364885.644068 trunc from 0x6a9930 to 0x8056c6 (0x15bd97 bytes) 000415[0] 1753364885.689963 trunc from 0x8056c7 to 0x29f148 (0x56657e bytes) 000416[1] 1753364885.711438 trunc from 0x29f149 to 0x0398b6 (0x265892 bytes) 000417[0] 1753364885.753871 read 0x017170 thru 0x018470 (0x1301 bytes) 000419[1] 1753364885.792941 mapwrite 0x1aa655 thru 0x1aa8a8 (0x254 bytes) 000421[1] 1753364885.850213 trunc from 0x1aa8a9 to 0x7f1d10 (0x647468 bytes) 000422[1] 1753364885.879246 trunc from 0x7f1d11 to 0x01805c (0x7d9cb4 bytes) 000424[0] 1753364885.910112 write 0x022e3b thru 0x0237da (0x9a0 bytes) 000425[0] 1753364885.923215 write 0x4c0834 thru 0x4c1cdd (0x14aa bytes) 000426[1] 1753364885.941716 read 0x2b3162 thru 0x2b341e (0x2bd bytes) 000428[1] 1753364885.962102 read 0x252005 thru 0x2532b8 (0x12b4 bytes) 000429[1] 1753364885.990564 mapwrite 0x686c2f thru 0x687bca (0xf9c bytes) 000432[0] 1753364886.056604 mapread 0x3d8cf1 thru 0x3d914e (0x45e bytes) 000437[0] 1753364886.169181 read 0x435632 thru 0x437384 (0x1d53 bytes) 000440[0] 1753364886.193351 trunc from 0x687bcb to 0x24be91 (0x43bd39 bytes) 000441[0] 1753364886.229253 mapwrite 0x94a9a8 thru 0x94c4c7 (0x1b20 bytes) 000443[1] 1753364886.302851 trunc from 0x94c4c8 to 0x811dad (0x13a71a bytes) 000445[0] 1753364886.324394 mapwrite 0x2265fc thru 0x2283de (0x1de3 bytes) 000450[1] 1753364886.423188 mapread 0x601c10 thru 0x603603 (0x19f4 bytes) 000451[0] 1753364886.441254 write 0x76c777 thru 0x76de9a (0x1724 bytes) 000452[0] 1753364886.458570 trunc from 0x811dae to 0x4b7880 (0x35a52d bytes) 000458[0] 1753364886.557605 read 0x1919e9 thru 0x19226e (0x886 bytes) 000459[0] 1753364886.562313 trunc from 0x4b7881 to 0x6f5629 (0x23dda9 bytes) 000461[1] 1753364886.602387 mapread 0x1ea0b0 thru 0x1eaa34 (0x985 bytes) 000462[1] 1753364886.625515 mapwrite 0x4db5f6 thru 0x4dcbc7 (0x15d2 bytes) 000464[1] 1753364886.712070 trunc from 0x6f562a to 0x546b48 (0x1aeae1 bytes) 000466[1] 1753364886.746441 mapread 0x0edc10 thru 0x0eef2f (0x1320 bytes) 000469[0] 1753364886.772069 read 0x4b909c thru 0x4b9a53 (0x9b8 bytes) 000471[1] 1753364886.805056 trunc from 0x546b49 to 0x1816aa (0x3c549e bytes) 000472[1] 1753364886.831790 mapread 0x01fbdb thru 0x020994 (0xdba bytes) 000474[0] 1753364886.837715 mapread 0x032b21 thru 0x032bc4 (0xa4 bytes) 000478[0] 1753364886.889580 mapwrite 0x450bfc thru 0x450d8b (0x190 bytes) 000479[1] 1753364886.966490 mapwrite 0x9b2c1d thru 0x9b3191 (0x575 bytes) 000482[0] 1753364887.035212 trunc from 0x9b3192 to 0x00f395 (0x9a3dfc bytes) 000484[1] 1753364887.093835 mapread 0x002b5f thru 0x004a5d (0x1eff bytes) 000487[0] 1753364887.145127 trunc from 0x00f396 to 0x8cd8cc (0x8be537 bytes) 000492[0] 1753364887.199375 mapread 0x4b4eec thru 0x4b51b3 (0x2c8 bytes) 000493[1] 1753364887.218904 read 0x828276 thru 0x828514 (0x29f bytes) 000498[0] 1753364887.262255 mapread 0x520b31 thru 0x520edb (0x3ab bytes) 000499[0] 1753364887.275677 trunc from 0x8cd8cd to 0x1b970d (0x7141bf bytes) 000500[1] 1753364887.305948 read 0x0fde29 thru 0x0fe218 (0x3f0 bytes) All operations completed A-OK! PASS 16b (13s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:48:09 (1753364889) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:48:11 (1753364891) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.20251 s, 41.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.15724 s, 53.3 MB/s ldlm.namespaces.MGC192.168.202.137@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff98bc4509a000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff98bc45ca2000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff98bc45ca2000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff98bc45ca2000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.5041 s, 23.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.32082 s, 79.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.924487 s, 113 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.13413 s, 49.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.35127 s, 44.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.23601 s, 84.8 MB/s PASS 16d (17s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:48:28 (1753364908) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.102339 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462042 s, 22.7 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:48:31 (1753364911) TIMEOUT PASS 16f (23s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:48:54 (1753364934) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 09:49:20 (1753364960) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 09:49:27 (1753364967) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 09:49:34 (1753364974) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.13478 s, 23.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.858281 s, 30.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.65043 s, 15.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.33557 s, 78.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.39386 s, 18.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.401916 s, 65.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5719 s, 16.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.511866 s, 51.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.666926 s, 39.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.31483 s, 19.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.37005 s, 19.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.987715 s, 26.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.02791 s, 25.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27397 s, 20.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.36124 s, 19.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.539001 s, 48.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.820531 s, 31.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7393 s, 15.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.80395 s, 14.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (20s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:49:54 (1753364994) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Jul 24 09:50:10 EDT 2025 + fsx 2 done: rc=0 Thu Jul 24 09:50:10 EDT 2025 + fsx 3 done: rc=0 Thu Jul 24 09:50:10 EDT 2025 + fsx 4 done: rc=0 Thu Jul 24 09:50:10 EDT 2025 + fsx 5 done: rc=0 Thu Jul 24 09:50:10 EDT 2025 PASS 16k (20s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:50:14 (1753365014) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 09:50:19 (1753365019) mmap test1: basic mmap operation (PASS, 0.072728s) mmap test2: MAP_PRIVATE not write back (PASS, 0.050868s) mmap test3: concurrent mmap ops on two nodes (PASS, 12.964s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.2081s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.090512s) mmap test6: check mmap write/read content on two nodes (PASS, 0.14191s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.50369s) mmap test11: multi-thread shared mmap (PASS, 0.54311s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0359s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:50:43 (1753365043) oleg237-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:50:45 (1753365045) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:50:48 (1753365048) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 09:50:51 (1753365051) now is 1753365053 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 new atime is 1753365114 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:51:56 (1753365116) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:51:59 (1753365119) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:52:01 (1753365121) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:52:04 (1753365124) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 09:52:06 (1753365126) utime: good mknod times 1753365126 <= 1753365127 <= 1753365127 for /mnt/lustre/f26a utime: good mknod times 1753365126 <= 1753365127 <= 1753365127 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 09:52:10 (1753365130) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:52:14 (1753365134) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-07-24 09:52:15.000000000 -0400 Birth: 2025-07-24 09:52:15.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-07-24 09:52:15.000000000 -0400 Birth: 2025-07-24 09:52:15.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:52:19 (1753365139) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:52:24 (1753365144) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:52:27 (1753365147) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg237-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.137@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 09:52:40 (1753365160) -rw-r--r-- 1 root root 0 Jul 24 09:52 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:52:46 (1753365166) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42818 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:52:50 (1753365170) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:52:54 (1753365174) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:52:56 (1753365176) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:52:57 (1753365177) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:52:58 (1753365178) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 09:53:00 (1753365180) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid in FULL state after 0 sec oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid in IDLE state after 0 sec oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid in FULL state after 0 sec oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid in IDLE state after 0 sec oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98bc4509a000.ost_server_uuid in IDLE state after 0 sec oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98bc4509a000.ost_server_uuid in IDLE state after 0 sec PASS 34 (96s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:54:36 (1753365276) fail_loc=0x80000317 Race attempt 0 Wait for 48934 49064 for 60 sec... - open/close 2246 (time 1753365288.52 total 10.00 last 224.55) total: 4000 open/close in 18.11 seconds: 220.88 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.137@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:55:41 (1753365341) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg237-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.433252 s, 121 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.45729 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.445294 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.431815 s, 121 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.392451 s, 134 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.460997 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.448074 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.470547 s, 111 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.428246 s, 122 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.440911 s, 119 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.447605 s, 117 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7444480) after(7495680) PASS 36 (233s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 09:59:34 (1753365574) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 - create 6279 (time 1753365586.35 total 10.00 last 627.85) total: 10000 create in 15.30 seconds: 653.51 ops/second PASS 37 (19s) == sanityn test 39a: file mtime does not change after rename ========================================================== 09:59:53 (1753365593) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 09:59:56 (1753365596) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 09:59:59 (1753365599) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 10:00:02 (1753365602) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 10:00:05 (1753365605) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 10:00:14 (1753365614) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 10:00:23 (1753365623) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 10:00:32 (1753365632) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 10:00:40 (1753365640) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 10:00:46 (1753365646) open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 10:00:52 (1753365652) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 10:00:58 (1753365658) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 10:01:03 (1753365663) Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 10:01:09 (1753365669) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 10:01:14 (1753365674) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 10:01:20 (1753365680) Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 10:01:25 (1753365685) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (889s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:16:14 (1753366574) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:16:20 (1753366580) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:16:25 (1753366585) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:16:31 (1753366591) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:16:36 (1753366596) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:16:42 (1753366602) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:16:48 (1753366608) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:16:53 (1753366613) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:16:59 (1753366619) PASS 43a (36s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:17:35 (1753366655) Conflict PASS 43b (11s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:17:46 (1753366666) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:18:00 (1753366680) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:18:14 (1753366694) Conflict PASS 43e (10s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:18:23 (1753366703) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (8s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:18:32 (1753366712) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (9s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:18:40 (1753366720) Conflict PASS 43h (8s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:18:48 (1753366728) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:18:50 (1753366730) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (75s) == sanityn test 43k: unlink vs create ==================== 10:20:05 (1753366805) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (711s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:31:56 (1753367516) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jul 24 10:32:01 EDT 2025 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:32:03 (1753367523) open: File exists Conflict PASS 44b (7s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:32:10 (1753367530) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:32:17 (1753367537) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:32:23 (1753367543) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:32:30 (1753367550) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:32:36 (1753367556) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:32:43 (1753367563) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:32:49 (1753367569) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:32:51 (1753367571) PASS 45a (41s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:33:32 (1753367612) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:33:38 (1753367618) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:33:45 (1753367625) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:33:51 (1753367631) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:33:56 (1753367636) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:34:02 (1753367642) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:34:08 (1753367648) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:34:13 (1753367653) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 10:34:14 (1753367654) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (567s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 10:43:41 (1753368221) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 10:43:47 (1753368227) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 10:43:52 (1753368232) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 10:43:58 (1753368238) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 10:44:04 (1753368244) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 10:44:10 (1753368250) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 10:44:16 (1753368256) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 10:44:22 (1753368262) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 10:44:28 (1753368268) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 10:44:29 (1753368269) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 10:44:31 (1753368271) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 10:44:32 (1753368272) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 10:44:33 (1753368273) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 10:44:34 (1753368274) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 10:44:36 (1753368276) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 10:44:37 (1753368277) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 10:44:38 (1753368278) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0109242 s, 937 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 10:44:46 (1753368286) 0+1 records in 0+1 records out 158 bytes copied, 0.0124139 s, 12.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260113) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 10:44:51 (1753368291) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00627058 s, 163 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305642 s, 34.3 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:45:06 (1753368306) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0062526 s, 164 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:45:14 (1753368314) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164605 s, 63.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:45:18 (1753368318) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2446 0x98e 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2446 0x98e 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 10:45:22 (1753368322) llite.lustre-ffff98bc4509a000.xattr_cache=1 llite.lustre-ffff98bc45ca2000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:23.000000000 -0400 Modify: 2025-07-24 10:45:23.000000000 -0400 Change: 2025-07-24 10:45:23.000000000 -0400 Birth: 2025-07-24 10:45:23.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:28.000000000 -0400 Modify: 2025-07-24 10:45:29.000000000 -0400 Change: 2025-07-24 10:45:29.000000000 -0400 Birth: 2025-07-24 10:45:29.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:34.000000000 -0400 Modify: 2025-07-24 10:45:34.000000000 -0400 Change: 2025-07-24 10:45:34.000000000 -0400 Birth: 2025-07-24 10:45:34.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:34.000000000 -0400 Modify: 2025-07-24 10:45:34.000000000 -0400 Change: 2025-07-24 10:45:34.000000000 -0400 Birth: 2025-07-24 10:45:34.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:40.000000000 -0400 Modify: 2025-07-24 10:45:40.000000000 -0400 Change: 2025-07-24 10:45:40.000000000 -0400 Birth: 2025-07-24 10:45:40.000000000 -0400 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 10:45:40.000000000 -0400 Modify: 2025-07-24 10:45:40.000000000 -0400 Change: 2025-07-24 10:45:40.000000000 -0400 Birth: 2025-07-24 10:45:40.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 10:45:47 (1753368347) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 10:45:55 (1753368355) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 10:46:04 (1753368364) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 10:46:17 (1753368377) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:46:27 (1753368387) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:46:29 (1753368389) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 10:46:42 (1753368402) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:46:57 (1753368417) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1753368418 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.69553 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (39s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 10:47:37 (1753368457) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1753368459 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.2993 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (43s) == sanityn test 60: Verify data_version behaviour ======== 10:48:20 (1753368500) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 10:48:22 (1753368502) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 10:48:25 (1753368505) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 10:48:29 (1753368509) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00377301 s, 17.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 10:48:31 (1753368511) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00731233 s, 9.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 10:48:32 (1753368512) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 10:48:33 (1753368513) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00618281 s, 10.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 10:48:35 (1753368515) llite.lustre-ffff98bc4509a000.xattr_cache=1 llite.lustre-ffff98bc45ca2000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 10:48:37 (1753368517) llite.lustre-ffff98bc4509a000.xattr_cache=1 llite.lustre-ffff98bc45ca2000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 10:48:40 (1753368520) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00633434 s, 162 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00754423 s, 136 kB/s 272891: get lock1 272890: get lock2 272890: done 272890: get lock1 272891: done 272891: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 272890: cannot get lock1: Resource deadlock avoided 272891: done 272891: exit rc=0 272890: done 272890: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 10:48:47 (1753368527) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0232115 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179677 s, 58.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273473 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 10:48:55 (1753368535) opening files: ........ closing files: ........ PASS 76 (25s) == sanityn test 77a: check FIFO NRS policy =============== 10:49:20 (1753368560) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146238 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354462 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345579 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102134 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102069 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879236 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853528 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106557 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108678 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110924 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116489 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117369 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110559 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117739 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114782 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114082 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115675 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314822 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030695 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343887 s, 30.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223389 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158951 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255438 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373258 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025929 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225208 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462506 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118815 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912828 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329485 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769144 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00600787 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776139 s, 135 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 10:49:23 (1753368563) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133284 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305758 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309531 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367898 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942262 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100428 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100713 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120318 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109351 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11039 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100174 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104207 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122225 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1159 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984378 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119927 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119022 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127717 s, 82.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674449 s, 155 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147146 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806997 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251366 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603846 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207542 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130927 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186993 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349886 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292528 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184259 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846847 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245818 s, 42.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942997 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593992 s, 177 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120893 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310892 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398973 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130822 s, 8.0 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158126 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158588 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19436 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182343 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19213 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178881 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192275 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18279 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195107 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212318 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188923 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207614 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229119 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128574 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182708 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216915 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205624 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344331 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106485 s, 98.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369074 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370782 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378732 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238785 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034842 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231156 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352452 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299313 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293391 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467977 s, 224 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 10:49:27 (1753368567) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153707 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287731 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455769 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433883 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037082 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342634 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847204 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914894 s, 11.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673292 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979779 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998813 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864611 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105711 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106892 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993752 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698493 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965969 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330064 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271135 s, 38.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315591 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314724 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240433 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230474 s, 45.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304347 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303729 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389173 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278876 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344007 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602486 s, 174 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222936 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131727 s, 79.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221153 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132854 s, 78.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153713 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349335 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486623 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936145 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124674 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992437 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10076 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115096 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149155 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144598 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144199 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14073 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114783 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113493 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126831 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169981 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151007 s, 6.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520039 s, 202 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199633 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213925 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242764 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164828 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364488 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448346 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220727 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331342 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748052 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247277 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183576 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342982 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230574 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030119 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352584 s, 29.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138156 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247185 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686185 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057029 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077017 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725011 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885291 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748133 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857598 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740451 s, 14.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146165 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158983 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301018 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250059 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366272 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182447 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196959 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197099 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305379 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110729 s, 94.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779635 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199925 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120946 s, 86.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178478 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233985 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942207 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243208 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140333 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754417 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197075 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226494 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161318 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104667 s, 100 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 10:49:33 (1753368573) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191543 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253802 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620596 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059995 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824385 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930114 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926618 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786475 s, 13.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100031 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771402 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918578 s, 11.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106544 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100425 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957495 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103128 s, 10.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963926 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12566 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297045 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325575 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208266 s, 50.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346953 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463203 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496043 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402312 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390758 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276569 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312596 s, 335 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490157 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180622 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168555 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246908 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011869 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878827 s, 119 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127081 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028894 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470735 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716675 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556458 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08296 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893868 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818133 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919296 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930392 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735775 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821319 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954238 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931042 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111459 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910094 s, 11.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094854 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00850029 s, 123 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232598 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201504 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028611 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218938 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246126 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243407 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289337 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283645 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325883 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309658 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251192 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274093 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302624 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347877 s, 30.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.177365 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278323 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421059 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508302 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413685 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051912 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824723 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841527 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102925 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118645 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966474 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119823 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128363 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122825 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107279 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132869 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143869 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016919 s, 62.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168952 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136761 s, 76.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158434 s, 66.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022773 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334231 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159435 s, 65.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227393 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022984 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223281 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245112 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281288 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311156 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265982 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259162 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303149 s, 34.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 10:49:39 (1753368579) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.37@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179608 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326536 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371014 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959017 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100145 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838118 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10756 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10164 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105735 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097981 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983759 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960466 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104679 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101244 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911682 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112349 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110496 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209275 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207071 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205512 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208318 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207791 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211402 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216143 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209919 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210462 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213663 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219531 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217434 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211278 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221284 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213088 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21182 s, 5.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135636 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035344 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270724 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496794 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496824 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647133 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633554 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656149 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831693 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052343 s, 20.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905183 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677701 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739847 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862759 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104228 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114641 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112053 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10274 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102778 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114526 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11322 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103991 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104886 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115122 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109281 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118709 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114976 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109825 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121563 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125322 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121775 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121409 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119714 s, 8.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142377 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296071 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275327 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789117 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616503 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917882 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898309 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987942 s, 10.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113756 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117183 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107875 s, 9.7 MB/sldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11351 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111988 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128484 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108377 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11232 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120388 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184001 s, 57.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184998 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283732 s, 37.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212543 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276868 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250381 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272591 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252113 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020008 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257373 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151336 s, 69.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129367 s, 81.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00729392 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070019 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412709 s, 254 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910414 s, 115 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14402 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318205 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026404 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125864 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131544 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132811 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137165 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141647 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133705 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120333 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147423 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140513 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142909 s, 7.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142894 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152879 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160603 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157533 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158658 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187281 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150796 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217532 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023048 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043163 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188925 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249573 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193421 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262287 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242077 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332654 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232378 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447837 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539697 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268295 s, 39.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:49:47 (1753368587) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.192336 s, 87.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031657 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382222 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389142 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436312 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531886 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977588 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805553 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816948 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122974 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918713 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987596 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130163 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125349 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970174 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997017 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107442 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151129 s, 69.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185307 s, 56.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207154 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166568 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211404 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204048 s, 51.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189913 s, 55.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265015 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196917 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240349 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214389 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228546 s, 45.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245449 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231103 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00966579 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522283 s, 201 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1779 s, 94.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540078 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562315 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870742 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832088 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762474 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879992 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101563 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988287 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903202 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809789 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107554 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116581 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105262 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111575 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940651 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134587 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987555 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131844 s, 79.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884959 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288243 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261143 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231078 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174799 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215842 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276394 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834104 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423649 s, 248 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023626 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327201 s, 320 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100158 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421137 s, 249 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005167 s, 203 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180859 s, 92.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438307 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642283 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587521 s, 17.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050948 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721816 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711148 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268027 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563242 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455557 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986134 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106539 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133999 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130186 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205144 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262584 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215412 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168438 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261392 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172179 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029356 s, 35.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354247 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173889 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161482 s, 64.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327007 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131613 s, 79.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287031 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026657 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057254 s, 183 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116719 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129282 s, 81.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105383 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166263 s, 63.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133807 s, 125 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285089 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399988 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395242 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069253 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914917 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976379 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848692 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086086 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983859 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701207 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120653 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101639 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114961 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885436 s, 11.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948784 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119605 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283894 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211509 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993821 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105479 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245199 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234959 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115817 s, 90.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270736 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334724 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317672 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204456 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333126 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217714 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221608 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309864 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356207 s, 29.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 10:49:57 (1753368597) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:50:01 (1753368601) oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg237-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg237-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 12 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg237-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 12 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg237-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 12 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 10:50:06 (1753368606) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:50:14 (1753368614) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143282 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351662 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773231 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776462 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834849 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768997 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694485 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651475 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914795 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885186 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861885 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818999 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079327 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968084 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106234 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922644 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105582 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215462 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207035 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190687 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297413 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00623152 s, 168 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255269 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171243 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496351 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224137 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378989 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312777 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427609 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309405 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070647 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312451 s, 336 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247075 s, 424 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04597 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4637 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (47s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:51:01 (1753368661) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465383 s, 36.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442115 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579753 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606199 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068694 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797523 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881081 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1055 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751014 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875216 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996674 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105112 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892581 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125354 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110169 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109107 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142326 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192072 s, 54.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188951 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215671 s, 48.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210362 s, 49.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223411 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203794 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236803 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179382 s, 58.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271644 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026623 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241596 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226787 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269191 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253378 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280239 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331054 s, 31.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7181 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9386 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26735 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132547 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140929 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135334 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161098 s, 6.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166772 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152651 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185102 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166377 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179771 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171204 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186649 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197242 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184547 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198659 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235658 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252782 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450181 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373777 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555876 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438814 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446745 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590928 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545887 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664782 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473954 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500165 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627932 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053518 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659387 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646656 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657108 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864043 s, 12.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9692 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9885 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (140s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:53:21 (1753368801) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25264 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676861 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630217 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598718 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761973 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103575 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798075 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926633 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124749 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144229 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144993 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174729 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184497 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177035 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154345 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192495 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223617 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712793 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059461 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585269 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573115 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660472 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792213 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047859 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784229 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657951 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833515 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644693 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651596 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690085 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069173 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901462 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768897 s, 13.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8297 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9575 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27631 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709538 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870334 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10222 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994998 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112838 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112873 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957773 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107412 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886486 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992982 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133339 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138782 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113646 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11625 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17441 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105151 s, 10.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311957 s, 33.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283537 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397431 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409594 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451875 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332296 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321886 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360331 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387655 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512327 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402435 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051379 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548765 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461491 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460764 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049363 s, 21.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9478 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8941 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (138s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:55:40 (1753368940) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31391 s, 53.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397175 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361348 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052322 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519284 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510906 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727749 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173837 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166571 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163143 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186136 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187681 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166496 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193605 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200095 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210165 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217011 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053059 s, 198 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601057 s, 174 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470855 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397284 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636775 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00297072 s, 353 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438274 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403849 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05494 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329713 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055926 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301595 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026613 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176406 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156087 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216743 s, 48.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.33141 s, 19.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85926 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.37@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.37@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.298487 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347753 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311476 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118915 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129346 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14458 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148212 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140356 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150915 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145672 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141195 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162437 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150941 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145594 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142164 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15527 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157514 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517205 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446539 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449421 s, 233 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00760224 s, 138 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414205 s, 25.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350067 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553968 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0056983 s, 184 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529463 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676623 s, 15.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425162 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409968 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389988 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564228 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367138 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145002 s, 72.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.98095 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88109 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.37@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.339571 s, 49.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302271 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378208 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100383 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110111 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996308 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117031 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125961 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111961 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116813 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110424 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119838 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132705 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124235 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126069 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123367 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166556 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822328 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228589 s, 459 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311326 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147403 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705881 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294978 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395685 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307926 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04828 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331563 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382285 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233847 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128269 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605609 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106455 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910806 s, 115 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47106 s, 19.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97421 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246083 s, 68.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196931 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413098 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394134 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615885 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838036 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165698 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17355 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194358 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182712 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199173 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196641 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202996 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227709 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191091 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239422 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259628 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620904 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138471 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161522 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152678 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152977 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181172 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383334 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119552 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108105 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088392 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102289 s, 10.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685633 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540642 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424003 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125664 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859117 s, 12.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99135 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86814 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.36183 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88276 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244436 s, 68.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482889 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313936 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513627 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100775 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948918 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122759 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13407 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145545 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159055 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15078 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141678 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150379 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14672 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158321 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178698 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166877 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161727 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544221 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482847 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588937 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475579 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841197 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804542 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00517826 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635277 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124852 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324522 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399685 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375348 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500982 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289554 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297092 s, 35.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7273 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5575 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355452 s, 47.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363578 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399521 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675092 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714776 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121931 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140163 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13408 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128839 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126259 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145002 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140244 s, 7.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119807 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130883 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136278 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124021 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139238 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028016 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358453 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285848 s, 36.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326029 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446723 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317063 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358436 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393456 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041946 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522973 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418508 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419585 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563597 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483499 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523965 s, 20.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26022 s, 19.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81947 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.242935 s, 69.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261536 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386029 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504492 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100476 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973797 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100213 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746232 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112198 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101735 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971972 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921118 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923832 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119872 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126243 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237374 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234482 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091953 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269212 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135624 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173534 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327937 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300324 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200047 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199274 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232581 s, 45.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233101 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022607 s, 46.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237664 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255628 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261042 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263633 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268485 s, 39.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg237-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg237-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8597 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76354 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84391 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76645 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (383s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 11:02:03 (1753369323) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (5s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 11:02:08 (1753369328) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg237-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.72048 s, 282 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg237-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0305 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 11:03:00 (1753369380) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161147 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374055 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359347 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661563 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064495 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114669 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98bc4509a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10983 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10681 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112451 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103717 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124999 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122311 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128998 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135317 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130305 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135288 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128844 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98bc4509a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226366 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321788 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364929 s, 287 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280479 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352437 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316761 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289886 s, 362 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026858 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360182 s, 29.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288834 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045687 s, 230 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223334 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175658 s, 59.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112034 s, 93.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134546 s, 77.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153143 s, 68.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02464 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97458 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.259487 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298257 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362984 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327747 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376756 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508154 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758574 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741234 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11186 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126884 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13545 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135821 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125457 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225568 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234408 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276327 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279852 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240235 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195011 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284543 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281485 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170316 s, 61.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214289 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131593 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174623 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159575 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233603 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214859 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249869 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228276 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275394 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028432 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312144 s, 33.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg237-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg237-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.10061 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00043 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (69s) == sanityn test 77o: Changing rank should not panic ====== 11:04:09 (1753369449) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 11:04:14 (1753369454) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (48s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 11:05:02 (1753369502) mds.MDS.mdt.nrs_policies=tbf oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg237-client: oleg237-server: ssh exited with exit code 36 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 oleg237-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg237-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 11:05:17 (1753369517) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 300651 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 11:05:57 (1753369557) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 11:06:01 (1753369561) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506196 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 11:06:02.000000000 -0400 Modify: 2025-07-24 11:06:02.000000000 -0400 Change: 2025-07-24 11:06:02.000000000 -0400 Birth: 2025-07-24 11:06:02.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 11:06:14 (1753369574) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 11:06:16 (1753369576) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 11:06:17 (1753369577) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 11:06:19 (1753369579) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 11:06:20 (1753369580) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 11:06:22 (1753369582) - open/close 2090 (time 1753369594.19 total 10.00 last 208.95) total: 2500 open/close in 11.94 seconds: 209.44 ops/second snapshot_time 1753369624.447535367 secs.nsecs start_time 1753369583.624692307 secs.nsecs elapsed_time 40.822843060 secs.nsecs open 2505 samples [usecs] 159 4646 1695171 1246847393 close 2504 samples [usecs] 81 5111 586612 179712186 mknod 2501 samples [usecs] 201 4446 1489016 965758152 unlink 2500 samples [usecs] 322 60641 2465621 11111173547 mkdir 2 samples [usecs] 959 1097 2056 2123090 rmdir 2 samples [usecs] 3528 10353 13881 119631393 rename 2500 samples [usecs] 277 10887 5438675 18089597757 getattr 3762 samples [usecs] 7 1862 115590 9540540 parallel_rename_file 2500 samples [usecs] 277 10887 5438675 18089597757 crossdir_rename 2500 samples [usecs] 277 10887 5438675 18089597757 PASS 81d (45s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 11:07:06 (1753369626) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7541 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7541 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 11:07:09 (1753369629) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 11:07:11 (1753369631) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 11:07:20 (1753369640) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1753369640 Finishing test test1 at 1753369641 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 11:07:23 (1753369643) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 11:07:25 (1753369645) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 11:07:27 (1753369647) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 11:07:28 (1753369648) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5610 0x15ea 0x240000400 1 5452 0x154c 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5611 0x15eb 0x240000400 1 5453 0x154d 0x280000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 11:07:38 (1753369658) osc.lustre-OST0000-osc-ffff98bc4509a000.idle_timeout=0 osc.lustre-OST0000-osc-ffff98bc45ca2000.idle_timeout=0 osc.lustre-OST0001-osc-ffff98bc4509a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff98bc45ca2000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0786662 s, 104 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08953 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5849: 317560 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0353132 s, 116 kB/s osc.lustre-OST0000-osc-ffff98bc4509a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff98bc45ca2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98bc4509a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98bc45ca2000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 11:07:49 (1753369669) llite.lustre-ffff98bc4509a000.fast_read=0 llite.lustre-ffff98bc45ca2000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0253194 s, 82.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff98bc4509a000.fast_read=1 llite.lustre-ffff98bc45ca2000.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 11:08:05 (1753369685) llite.lustre-ffff98bc4509a000.fast_read=0 llite.lustre-ffff98bc45ca2000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00325913 s, 3.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff98bc4509a000.fast_read=1 llite.lustre-ffff98bc45ca2000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 11:08:14 (1753369694) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 11:08:15 (1753369695) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00482611 s, 849 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 11:08:18 (1753369698) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0374001 s, 56.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 11:08:21 (1753369701) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0422247 s, 49.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 11:08:24 (1753369704) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00259334 s, 4.6 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 11:08:27 (1753369707) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000688979 s, 5.9 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 11:08:30 (1753369710) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000747477 s, 5.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 11:08:33 (1753369713) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000833754 s, 4.9 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 11:08:37 (1753369717) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1753369718 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1753369721 Finishing test test23 at 1753369721 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 11:08:45 (1753369725) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 11:08:46 (1753369726) fail_loc=0x416 PASS 105 (19s) == sanityn test 106a: Verify the btime via statx() ======= 11:09:05 (1753369745) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 11:09:06 (1753369746) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00441958 s, 232 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508731 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 11:09:07.000000000 -0400 Modify: 2025-07-24 11:09:07.000000000 -0400 Change: 2025-07-24 11:09:07.000000000 -0400 Birth: 2025-07-24 11:09:07.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272508731 -rw-r--r-- 1 0 0 1753369747 1753369747 1753369747 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508731 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 11:09:07.000000000 -0400 Modify: 2025-07-24 11:09:07.000000000 -0400 Change: 2025-07-24 11:09:07.000000000 -0400 Birth: 2025-07-24 11:09:07.000000000 -0400 1753369747 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 11:09:09 (1753369749) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 11:09:12 (1753369752) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.108646 s, 96.5 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5459 0x1553 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 11:09:16 (1753369756) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.102094 s, 103 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5620 0x15f4 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7541 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 11:09:25 (1753369765) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00275516 s, 3.6 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.35167 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 11:09:32 (1753369772) 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre 192.168.202.137@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 PASS 109 (599s) == sanityn test 110: do not grant another lock on resend ========================================================== 11:19:31 (1753370371) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 11:19:32.000000000 -0400 Modify: 2025-07-24 11:19:32.000000000 -0400 Change: 2025-07-24 11:19:32.000000000 -0400 Birth: 2025-07-24 11:19:32.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg237-client.virtnet oleg237-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 11:20:36 (1753370436) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 11:20:38 (1753370438) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 11:20:39 (1753370439) lustre-OST0000-osc-ffff98bc433ed000 active. lustre-OST0001-osc-ffff98bc433ed000 active. lustre-OST0000-osc-ffff98bc433ed000 active. lustre-OST0001-osc-ffff98bc433ed000 active. MGC192.168.202.137@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 11:20:41 (1753370441) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 11:20:43 (1753370443) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 11:20:44 (1753370444) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 11:20:45 (1753370445) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=19 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0904 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0978 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.095 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0987 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0869 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0963 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0923 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0895 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.093 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1121 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.149 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0996 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1276 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1521 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1163 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1337 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1552 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1421 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2267 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1951 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.202 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1771 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2506 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.216 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2183 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1793 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2348 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2383 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2493 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2541 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2911 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3447 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2917 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2711 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2819 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2895 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.378 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.357 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98bc433ed000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98bc433ed000.ost_server_uuid in FULL state after 0 sec PASS 200 (107s) cleanup: ====================================================== == sanityn test complete, duration 5982 sec ============== 11:22:32 (1753370552) === sanityn: start cleanup 11:22:33 (1753370553) === Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg237-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 11:23:25 (1753370605) ===