-----============= acceptance-small: sanityn ============----- Mon Dec 22 02:24:34 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg217-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 02:24:42 (1766388282) === Starting client oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Mount client oleg217-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Started clients oleg217-client.virtnet: 192.168.202.117@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg217-client.virtnet: executing check_config_client /mnt/lustre oleg217-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg217-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff969d48441000.idle_timeout=debug osc.lustre-OST0000-osc-ffff969d59987800.idle_timeout=debug osc.lustre-OST0001-osc-ffff969d48441000.idle_timeout=debug osc.lustre-OST0001-osc-ffff969d59987800.idle_timeout=debug disable quota as required oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 02:24:56 (1766388296) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217675 s, 48.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7549] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:24:57 (1766388297) PASS 0 (4s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:25:01 (1766388301) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:25:04 (1766388304) -rw-r--r-- 1 root root 0 Dec 22 02:25 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (3s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:25:07 (1766388307) -rw-r--r-- 1 root root 0 Dec 22 02:25 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:25:10 (1766388310) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:25:13 (1766388313) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (3s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:25:16 (1766388316) total 1 -rwxrwxrwx 1 root root 0 Dec 22 02:25 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:25 f2b total 1 -rwxrwxrwx 1 root root 0 Dec 22 02:25 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:25 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00688377 s, 74.4 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:25:19 (1766388319) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 02:25:20 (1766388320) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165808 s, 12.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:25:24 (1766388324) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:25:27 (1766388327) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:25:31 (1766388331) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 02:25:34 (1766388334) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:25:37 (1766388337) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:25:40 (1766388340) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (3s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:25:43 (1766388343) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 02:25:46 (1766388346) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221248 s, 47.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0172123 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:25:48 (1766388348) 1+0 records in 1+0 records out 1 byte copied, 0.00297089 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00883631 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00472816 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00548231 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00453988 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00486613 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00406199 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00642569 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00696448 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00647505 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00473373 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00427589 s, 0.2 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:25:51 (1766388351) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00393245 s, 781 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0136434 s, 300 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0002182 s, 14.1 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:25:54 (1766388354) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:25:56 (1766388356) warning: '-runtime' deprecated, use '-t runtime' instead - link 1020 (time 1766388367.82 total 10.01 last 101.89) - link 1466 (time 1766388377.85 total 20.04 last 44.47) - link 1857 (time 1766388387.87 total 30.06 last 39.01) - link 2246 (time 1766388397.88 total 40.07 last 38.86) - link 2600 (time 1766388407.91 total 50.10 last 35.29) - link 3027 (time 1766388417.92 total 60.12 last 42.65) - link 3706 (time 1766388427.93 total 70.13 last 67.82) - link 4360 (time 1766388437.94 total 80.14 last 65.34) - link 5119 (time 1766388447.95 total 90.15 last 75.82) total: 5898 link in 100.01 seconds: 58.97 ops/second using seed 3305781987 running for 100 seconds - stat 10000 (time 1766388367 ; total 9 ; last 9) - stat 18521 (time 1766388378 ; total 20 ; last 11) - stat 20000 (time 1766388379 ; total 21 ; last 1) - stat 26486 (time 1766388390 ; total 32 ; last 11) - stat 30000 (time 1766388395 ; total 37 ; last 5) - stat 36986 (time 1766388406 ; total 48 ; last 11) - stat 40000 (time 1766388413 ; total 55 ; last 7) - stat 45391 (time 1766388424 ; total 66 ; last 11) - stat 50000 (time 1766388433 ; total 75 ; last 9) - stat 55385 (time 1766388444 ; total 86 ; last 11) - stat 60000 (time 1766388453 ; total 95 ; last 9) total: 62655 stats in 100 seconds: 626.549988 stats/second - unlinked 0 (time 1766388408 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile5898) error: No such file or directory total: 5897 unlinks in 90 seconds: 65.522224 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21326) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21328) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 02:28:21 (1766388501) f13 f13 PASS 13 (4s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:28:25 (1766388505) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22512 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (4s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:28:29 (1766388509) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23093 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:28:32 (1766388512) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23670 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:28:35 (1766388515) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24252 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:28:38 (1766388518) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24834 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:28:42 (1766388522) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7526400 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7526400KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (45s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 02:29:28 (1766388568) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.340368 s, 24.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.304431 s, 27.6 MB/s Chance of close/open is 1 in 50 Seed set to 8571 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1e6d88 000001[0] 1766388571.643070 trunc from 00000000 to 0x1e6d87 (0x1e6d88 bytes) 000004[1] 1766388571.711971 write 0x6d5de3 thru 0x6d8776 (0x2994 bytes) 000006[1] 1766388571.748883 mapwrite 0x2a61f2 thru 0x2ab8aa (0x56b9 bytes) 000010[0] 1766388571.848844 mapwrite 0x2efc3d thru 0x2f3abd (0x3e81 bytes) 000011[0] 1766388572.020385 write 0x6a7e5b thru 0x6ac1a6 (0x434c bytes) 000013[1] 1766388572.075573 mapread 0x27ba8a thru 0x27c47d (0x9f4 bytes) 000017[1] 1766388572.143459 mapread 0x54207d thru 0x542c8a (0xc0e bytes) 000023[0] 1766388572.179049 mapwrite 0x2584a2 thru 0x25bd26 (0x3885 bytes) 000026[1] 1766388572.276395 read 0x358245 thru 0x35b5c7 (0x3383 bytes) truncating to largest ever: 0x82f1c4 000028[0] 1766388572.313596 trunc from 0x6d8777 to 0x82f1c3 (0x156a4d bytes) 000029[0] 1766388572.389720 read 0x62fc63 thru 0x63f688 (0xfa26 bytes) 000030[0] 1766388572.430295 read 0x5278be thru 0x52e76e (0x6eb1 bytes) 000031[0] 1766388572.472122 write 0x8d7cdf thru 0x8dfa9e (0x7dc0 bytes) 000032[1] 1766388572.497573 read 0x24e84d thru 0x259d40 (0xb4f4 bytes) 000034[0] 1766388572.537244 read 0x7e1329 thru 0x7eb4d6 (0xa1ae bytes) 000035[0] 1766388572.558118 trunc from 0x8dfa9f to 0x09e737 (0x841367 bytes) 000037[1] 1766388572.589767 trunc from 0x09e738 to 0x7d26ea (0x733fb3 bytes) 000038[0] 1766388572.669230 mapread 0x281f5c thru 0x29047a (0xe51f bytes) 000041[0] 1766388572.737267 mapread 0x5332e4 thru 0x543272 (0xff8f bytes) 000042[0] 1766388572.800316 mapwrite 0x754281 thru 0x75a371 (0x60f1 bytes) 000043[0] 1766388572.886566 trunc from 0x7d26eb to 0x0ae9c6 (0x723d24 bytes) 000044[1] 1766388572.941301 mapread 0x006f5f thru 0x0124af (0xb551 bytes) 000045[0] 1766388573.040778 trunc from 0x0ae9c7 to 0x5b088f (0x501ec9 bytes) 000048[1] 1766388573.123055 mapread 0x3e7ce6 thru 0x3ea459 (0x2774 bytes) 000052[0] 1766388573.203787 read 0x0217ac thru 0x028763 (0x6fb8 bytes) 000056[1] 1766388573.238574 trunc from 0x5b0890 to 0x5d50a8 (0x24819 bytes) 000057[1] 1766388573.282483 mapwrite 0x3ddbca thru 0x3e1174 (0x35ab bytes) 000061[0] 1766388573.363472 mapread 0x0cff75 thru 0x0dd08a (0xd116 bytes) 000063[1] 1766388573.419462 mapread 0x2b6664 thru 0x2b7ee0 (0x187d bytes) 000064[1] 1766388573.433219 read 0x2e67b0 thru 0x2ee788 (0x7fd9 bytes) 000065[1] 1766388573.484447 read 0x23b4d2 thru 0x23d630 (0x215f bytes) 000066[1] 1766388573.520188 trunc from 0x5d50a9 to 0x4c5d42 (0x10f366 bytes) 000067[1] 1766388573.583687 read 0x14ef53 thru 0x15af2a (0xbfd8 bytes) 000068[1] 1766388573.618901 read 0x22a33e thru 0x2369d2 (0xc695 bytes) 000069[1] 1766388573.633965 write 0x1242cc thru 0x12c1e5 (0x7f1a bytes) 000070[1] 1766388573.674211 mapwrite 0x4efb36 thru 0x4f9dca (0xa295 bytes) 000072[1] 1766388573.787987 trunc from 0x4f9dcb to 0x4abec5 (0x4df05 bytes) 000073[0] 1766388573.812268 mapwrite 0x54516b thru 0x548106 (0x2f9c bytes) 000074[0] 1766388573.952307 mapwrite 0x3d114b thru 0x3d7941 (0x67f7 bytes) 000075[0] 1766388574.060602 trunc from 0x548107 to 0x1963f3 (0x3b1d13 bytes) 000076[1] 1766388574.098253 write 0x28486f thru 0x29291b (0xe0ad bytes) 000077[0] 1766388574.137222 read 0x0f541a thru 0x0feb9e (0x9785 bytes) 000078[0] 1766388574.211931 trunc from 0x29291c to 0x6d21f2 (0x43f8d7 bytes) 000080[0] 1766388574.278232 write 0x91d889 thru 0x926845 (0x8fbd bytes) 000081[0] 1766388574.281828 write 0x1efadc thru 0x1f3b5f (0x4084 bytes) 000082[1] 1766388574.361249 mapwrite 0x5263bf thru 0x529062 (0x2ca4 bytes) 000086[1] 1766388574.544573 trunc from 0x926846 to 0x73ec2d (0x1e7c18 bytes) 000088[1] 1766388574.614100 mapread 0x4cbc3a thru 0x4cd28a (0x1651 bytes) 000089[0] 1766388574.631471 read 0x1f06a1 thru 0x1f4652 (0x3fb2 bytes) 000090[1] 1766388574.670094 mapwrite 0x4cf147 thru 0x4d1877 (0x2731 bytes) 000092[0] 1766388574.766597 mapwrite 0x77c08f thru 0x78372d (0x769f bytes) 000096[1] 1766388574.954356 mapread 0x4ef79c thru 0x4faf43 (0xb7a8 bytes) 000098[1] 1766388575.024981 trunc from 0x78372e to 0x00fb41 (0x773bec bytes) 000102[0] 1766388575.097863 trunc from 0x00fb42 to 0x285524 (0x2759e3 bytes) 000103[0] 1766388575.158590 read 0x01c59b thru 0x02544b (0x8eb1 bytes) 000104[1] 1766388575.254380 mapread 0x14c2f6 thru 0x156ffa (0xad05 bytes) 000106[0] 1766388575.339660 trunc from 0x285525 to 0x62ad3f (0x3a581b bytes) 000108[0] 1766388575.402562 mapread 0x0e3e90 thru 0x0e86ce (0x483f bytes) 000109[0] 1766388575.473365 write 0x0a52a2 thru 0x0aeadd (0x983c bytes) 000110[0] 1766388575.486106 write 0x6de4a8 thru 0x6ebaf2 (0xd64b bytes) 000112[0] 1766388575.513384 trunc from 0x6ebaf3 to 0x761aeb (0x75ff9 bytes) 000113[0] 1766388575.576460 read 0x6ebec9 thru 0x6f4cae (0x8de6 bytes) 000115[1] 1766388575.615780 trunc from 0x761aec to 0x48f3d1 (0x2d271a bytes) 000118[1] 1766388575.718076 mapwrite 0x592880 thru 0x59feb2 (0xd633 bytes) 000119[1] 1766388575.814858 trunc from 0x59feb3 to 0x42cbde (0x1732d4 bytes) 000123[1] 1766388575.920844 mapwrite 0x2db02f thru 0x2e9939 (0xe90b bytes) 000124[0] 1766388575.990247 write 0x613b88 thru 0x61b08e (0x7507 bytes) 000126[1] 1766388576.046713 read 0x12edc5 thru 0x13635f (0x759b bytes) 000128[1] 1766388576.108050 mapwrite 0x09ab0b thru 0x09c666 (0x1b5c bytes) 000130[0] 1766388576.247633 write 0x3a6fb5 thru 0x3aa482 (0x34ce bytes) 000131[1] 1766388576.311977 mapwrite 0x0394a1 thru 0x0468a1 (0xd401 bytes) 000132[1] 1766388576.503220 mapwrite 0x3102c1 thru 0x31fe7d (0xfbbd bytes) 000133[0] 1766388576.621381 write 0x45b5a8 thru 0x466cb7 (0xb710 bytes) 000138[0] 1766388576.758580 write 0x789a8b thru 0x78ed03 (0x5279 bytes) 000139[0] 1766388576.778004 write 0x1c0326 thru 0x1c5429 (0x5104 bytes) 000140[0] 1766388576.797442 read 0x4431d7 thru 0x44a64b (0x7475 bytes) 000143[0] 1766388576.858044 mapread 0x1d4372 thru 0x1d57ad (0x143c bytes) 000144[0] 1766388576.886408 mapread 0x499129 thru 0x4a6539 (0xd411 bytes) 000146[0] 1766388576.946223 trunc from 0x78ed04 to 0x1a388a (0x5eb479 bytes) 000148[1] 1766388576.998473 mapread 0x0be2a4 thru 0x0c903d (0xad9a bytes) 000150[0] 1766388577.101155 trunc from 0x1a388b to 0x1d7b95 (0x3430b bytes) 000151[0] 1766388577.150618 write 0x4eb4ba thru 0x4ebab8 (0x5ff bytes) 000152[0] 1766388577.174584 read 0x3016b5 thru 0x306318 (0x4c64 bytes) 000157[1] 1766388577.265334 read 0x44f451 thru 0x458b11 (0x96c1 bytes) 000158[0] 1766388577.307185 trunc from 0x4ebab9 to 0x6196de (0x12dc26 bytes) 000162[0] 1766388577.389134 write 0x3e64a9 thru 0x3f3703 (0xd25b bytes) 000163[0] 1766388577.403669 write 0x6fc712 thru 0x700c79 (0x4568 bytes) 000164[1] 1766388577.404896 read 0x264a60 thru 0x26bde0 (0x7381 bytes) 000165[0] 1766388577.443417 mapwrite 0x45270d thru 0x45a80c (0x8100 bytes) 000167[0] 1766388577.513008 read 0x10544f thru 0x107d61 (0x2913 bytes) 000169[0] 1766388577.519413 trunc from 0x700c7a to 0x0afc1d (0x65105c bytes) 000170[1] 1766388577.568555 mapwrite 0x2e87e6 thru 0x2f6399 (0xdbb4 bytes) 000173[0] 1766388577.722942 read 0x20f1cb thru 0x215a9a (0x68d0 bytes) 000176[1] 1766388577.852918 mapread 0x275d5e thru 0x2813d4 (0xb677 bytes) 000177[1] 1766388577.871983 mapwrite 0x189177 thru 0x18a542 (0x13cc bytes) 000178[1] 1766388577.979160 mapread 0x227a86 thru 0x22b806 (0x3d81 bytes) 000179[0] 1766388578.012062 read 0x25b392 thru 0x260a6e (0x56dd bytes) 000180[0] 1766388578.092194 write 0x028b75 thru 0x02bc81 (0x310d bytes) 000181[0] 1766388578.127261 write 0x4342f7 thru 0x4394a0 (0x51aa bytes) 000182[0] 1766388578.137250 trunc from 0x4394a1 to 0x81c8e8 (0x3e3448 bytes) 000192[1] 1766388578.205719 trunc from 0x81c8e9 to 0x6c5df2 (0x156af6 bytes) 000193[0] 1766388578.274875 write 0x264700 thru 0x2724f6 (0xddf7 bytes) 000195[1] 1766388578.388150 mapread 0x6b132c thru 0x6b44f0 (0x31c5 bytes) 000196[1] 1766388578.448141 write 0x692c35 thru 0x69a271 (0x763d bytes) 000199[1] 1766388578.517364 read 0x25e862 thru 0x2688cf (0xa06e bytes) 000201[1] 1766388578.576904 mapwrite 0x529310 thru 0x532449 (0x913a bytes) 000203[1] 1766388578.669924 trunc from 0x6c5df3 to 0x789580 (0xc378e bytes) 000204[0] 1766388578.748274 write 0x2c7331 thru 0x2cc1d7 (0x4ea7 bytes) 000206[1] 1766388578.813677 write 0x42d00a thru 0x438155 (0xb14c bytes) 000207[1] 1766388578.834308 mapwrite 0x74c689 thru 0x757f64 (0xb8dc bytes) 000208[1] 1766388578.957795 write 0x0d72f5 thru 0x0de1d3 (0x6edf bytes) 000210[1] 1766388579.012256 trunc from 0x789581 to 0x200af1 (0x588a8f bytes) 000211[1] 1766388579.031861 trunc from 0x200af2 to 0x286cb4 (0x861c3 bytes) 000212[0] 1766388579.053822 mapread 0x159ac4 thru 0x161c50 (0x818d bytes) 000214[1] 1766388579.134109 mapwrite 0x936bc2 thru 0x938161 (0x15a0 bytes) 000216[0] 1766388579.268798 trunc from 0x938162 to 0x0f6a1a (0x841747 bytes) 000217[1] 1766388579.331304 mapread 0x021dc5 thru 0x0236fe (0x193a bytes) 000219[0] 1766388579.390162 write 0x4de5ab thru 0x4e6948 (0x839e bytes) 000220[0] 1766388579.406341 trunc from 0x4e6949 to 0x040857 (0x4a60f1 bytes) 000221[1] 1766388579.459112 mapread 0x019bb5 thru 0x01fa36 (0x5e82 bytes) 000227[0] 1766388579.554252 mapwrite 0x4d64fb thru 0x4d86e0 (0x21e6 bytes) 000228[1] 1766388579.669105 read 0x1a5057 thru 0x1b3775 (0xe71f bytes) 000229[0] 1766388579.729221 mapread 0x3cc695 thru 0x3d44c9 (0x7e35 bytes) 000232[0] 1766388579.807045 read 0x2f575f thru 0x2f70a2 (0x1944 bytes) 000235[1] 1766388579.816204 mapread 0x495877 thru 0x4a1a40 (0xc1ca bytes) 000237[1] 1766388579.852093 read 0x489ade thru 0x48af66 (0x1489 bytes) 000242[0] 1766388579.861520 mapread 0x1b65a7 thru 0x1c4a55 (0xe4af bytes) 000243[1] 1766388579.895631 read 0x3c4208 thru 0x3d4193 (0xff8c bytes) 000245[1] 1766388579.896477 read 0x40b786 thru 0x40cdb3 (0x162e bytes) 000246[1] 1766388579.897724 write 0x4e0639 thru 0x4e0b53 (0x51b bytes) 000247[0] 1766388579.917413 write 0x757dc5 thru 0x75b910 (0x3b4c bytes) truncating to largest ever: 0x9b168c 000248[1] 1766388579.949777 trunc from 0x75b911 to 0x9b168b (0x255d7b bytes) 000249[0] 1766388580.054473 read 0x93d66c thru 0x93ea2d (0x13c2 bytes) 000251[1] 1766388580.087329 trunc from 0x9b168c to 0x7eab7a (0x1c6b11 bytes) 000252[1] 1766388580.125392 write 0x0a2b6d thru 0x0a8163 (0x55f7 bytes) 000255[0] 1766388580.151702 trunc from 0x7eab7b to 0x252dd4 (0x597da6 bytes) 000256[1] 1766388580.192957 mapread 0x111627 thru 0x11e444 (0xce1e bytes) 000260[0] 1766388580.289442 trunc from 0x252dd5 to 0x4fb089 (0x2a82b5 bytes) 000261[0] 1766388580.335833 write 0x410e17 thru 0x4208e7 (0xfad1 bytes) 000263[0] 1766388580.364959 write 0x878316 thru 0x885e3b (0xdb26 bytes) 000264[1] 1766388580.367133 read 0x05acf2 thru 0x06a303 (0xf612 bytes) 000265[0] 1766388580.434198 mapread 0x0185c4 thru 0x0185c7 (0x4 bytes) 000266[1] 1766388580.465236 read 0x48c2f1 thru 0x49ba0a (0xf71a bytes) 000267[1] 1766388580.545159 mapwrite 0x124550 thru 0x129854 (0x5305 bytes) 000268[0] 1766388580.725424 mapread 0x25dc8d thru 0x25f020 (0x1394 bytes) 000270[0] 1766388580.776631 mapread 0x065bb0 thru 0x06c048 (0x6499 bytes) 000273[0] 1766388580.799081 mapread 0x544db4 thru 0x5483f4 (0x3641 bytes) 000274[1] 1766388580.828839 mapwrite 0x457b3a thru 0x45e10e (0x65d5 bytes) 000278[0] 1766388580.912795 write 0x481b2d thru 0x483d60 (0x2234 bytes) 000279[0] 1766388580.932134 trunc from 0x885e3c to 0x121769 (0x7646d2 bytes) 000282[0] 1766388580.979168 trunc from 0x12176a to 0x45d42a (0x33bcc1 bytes) 000286[0] 1766388581.053094 write 0x8464cf thru 0x85246c (0xbf9e bytes) 000287[0] 1766388581.057275 mapwrite 0x7fed85 thru 0x80be58 (0xd0d4 bytes) 000289[0] 1766388581.219803 mapwrite 0x42cb6d thru 0x437b62 (0xaff6 bytes) 000290[0] 1766388581.338659 mapwrite 0x739ad2 thru 0x744837 (0xad66 bytes) 000291[1] 1766388581.441577 write 0x82963a thru 0x833666 (0xa02d bytes) 000292[1] 1766388581.488447 read 0x5e6afc thru 0x5edd37 (0x723c bytes) 000293[1] 1766388581.527341 mapread 0x4abce7 thru 0x4ad79c (0x1ab6 bytes) 000295[0] 1766388581.600863 mapread 0x3c4a75 thru 0x3ca40c (0x5998 bytes) 000298[1] 1766388581.662226 read 0x036fce thru 0x0459ef (0xea22 bytes) 000300[1] 1766388581.673592 write 0x816067 thru 0x8229d0 (0xc96a bytes) 000301[0] 1766388581.708055 mapwrite 0x6491a7 thru 0x6504c8 (0x7322 bytes) 000302[0] 1766388581.855111 read 0x53e62a thru 0x53f795 (0x116c bytes) 000303[0] 1766388581.873360 write 0x71349e thru 0x71f133 (0xbc96 bytes) 000306[1] 1766388581.966438 read 0x56da44 thru 0x570d6f (0x332c bytes) 000307[0] 1766388582.005372 mapread 0x4015c5 thru 0x40fc38 (0xe674 bytes) 000309[1] 1766388582.103046 trunc from 0x85246d to 0x4cb4ed (0x386f7f bytes) 000310[1] 1766388582.147211 trunc from 0x4cb4ee to 0x94903e (0x47db51 bytes) 000311[0] 1766388582.187719 read 0x675414 thru 0x678176 (0x2d63 bytes) 000312[0] 1766388582.227109 mapwrite 0x5d3f5a thru 0x5df925 (0xb9cc bytes) 000315[1] 1766388582.348623 write 0x9a45d7 thru 0x9a8a54 (0x447e bytes) 000317[1] 1766388582.394424 mapwrite 0x191783 thru 0x194d97 (0x3615 bytes) 000319[0] 1766388582.547390 read 0x2ae96a thru 0x2b642d (0x7ac4 bytes) 000321[0] 1766388582.586934 mapwrite 0x90073c thru 0x90476a (0x402f bytes) 000323[1] 1766388582.702835 mapread 0x6b6a16 thru 0x6bbf83 (0x556e bytes) 000324[1] 1766388582.763194 mapread 0x180a79 thru 0x18ef19 (0xe4a1 bytes) 000325[0] 1766388582.807222 mapwrite 0x3479de thru 0x34bdec (0x440f bytes) 000326[0] 1766388582.937688 mapread 0x607ca6 thru 0x60e2df (0x663a bytes) 000327[0] 1766388582.966364 trunc from 0x9a8a55 to 0x010418 (0x99863c bytes) 000328[0] 1766388582.999114 mapwrite 0x0c7868 thru 0x0ce241 (0x69da bytes) 000330[0] 1766388583.101393 mapwrite 0x8dfe2e thru 0x8ecab9 (0xcc8c bytes) 000331[0] 1766388583.225605 mapread 0x40aabd thru 0x419270 (0xe7b4 bytes) 000332[0] 1766388583.273405 read 0x44da31 thru 0x45ac86 (0xd256 bytes) 000333[0] 1766388583.287344 trunc from 0x8ecaba to 0x6f17ca (0x1fb2ef bytes) 000334[0] 1766388583.309591 write 0x587001 thru 0x592bba (0xbbba bytes) 000335[0] 1766388583.367783 read 0x04fc88 thru 0x05383d (0x3bb6 bytes) 000336[0] 1766388583.392539 write 0x97dc48 thru 0x981074 (0x342d bytes) 000338[0] 1766388583.416398 mapwrite 0x4fc5b3 thru 0x503981 (0x73cf bytes) 000339[0] 1766388583.520165 write 0x5f561b thru 0x5f75ae (0x1f94 bytes) 000340[0] 1766388583.539824 mapread 0x6f8216 thru 0x703d74 (0xbb5f bytes) 000342[1] 1766388583.619942 write 0x00af4e thru 0x0115c3 (0x6676 bytes) 000346[1] 1766388583.717087 trunc from 0x981075 to 0x3c63b1 (0x5bacc3 bytes) 000347[0] 1766388583.777954 write 0x36f8e3 thru 0x36fa1c (0x13a bytes) 000348[1] 1766388583.823048 mapread 0x12627f thru 0x1361db (0xff5d bytes) 000349[0] 1766388583.883591 trunc from 0x3c63b2 to 0x7f0568 (0x42a1b7 bytes) 000350[0] 1766388583.931305 mapwrite 0x8f718f thru 0x8fbf94 (0x4e06 bytes) 000351[1] 1766388584.033537 write 0x8284f4 thru 0x8350d5 (0xcbe2 bytes) 000352[1] 1766388584.082086 trunc from 0x8fbf95 to 0x5e8eaf (0x3130e5 bytes) 000356[1] 1766388584.150295 trunc from 0x5e8eb0 to 0x34b926 (0x29d589 bytes) 000357[0] 1766388584.207412 read 0x217eed thru 0x221300 (0x9414 bytes) 000358[0] 1766388584.278406 read 0x28e130 thru 0x292fb1 (0x4e82 bytes) 000360[0] 1766388584.301155 write 0x53d1ca thru 0x541363 (0x419a bytes) 000362[0] 1766388584.337420 write 0x7952d3 thru 0x796ba6 (0x18d4 bytes) 000364[0] 1766388584.339516 trunc from 0x796ba7 to 0x7b4efe (0x1e358 bytes) 000365[0] 1766388584.396373 trunc from 0x7b4eff to 0x941862 (0x18c964 bytes) 000366[1] 1766388584.436583 trunc from 0x941863 to 0x288ab3 (0x6b8daf bytes) 000367[1] 1766388584.504577 write 0x4f27df thru 0x4fff47 (0xd769 bytes) 000369[1] 1766388584.554658 trunc from 0x4fff48 to 0x1849da (0x37b56d bytes) 000370[0] 1766388584.592613 trunc from 0x1849db to 0x90b38b (0x7869b1 bytes) 000374[1] 1766388584.689872 trunc from 0x90b38c to 0x7b9e28 (0x151563 bytes) 000375[1] 1766388584.724474 mapwrite 0x8f7a56 thru 0x8fb37f (0x392a bytes) 000376[0] 1766388584.844120 write 0x450e25 thru 0x460d69 (0xff45 bytes) 000377[1] 1766388584.879114 trunc from 0x8fb380 to 0x611f38 (0x2e9447 bytes) 000378[1] 1766388584.910834 mapread 0x0b7b0b thru 0x0c5859 (0xdd4f bytes) 000380[0] 1766388584.957134 mapwrite 0x1e485e thru 0x1eef40 (0xa6e3 bytes) 000382[0] 1766388585.076017 read 0x3131f7 thru 0x31ef55 (0xbd5f bytes) 000385[1] 1766388585.121923 mapwrite 0x5cd84b thru 0x5cf88f (0x2045 bytes) 000389[0] 1766388585.218275 trunc from 0x611f39 to 0x1d9e7e (0x4380ba bytes) 000391[0] 1766388585.271229 trunc from 0x1d9e7f to 0x4bc1e3 (0x2e2365 bytes) 000393[1] 1766388585.288970 mapread 0x3426ba thru 0x346428 (0x3d6f bytes) 000394[1] 1766388585.328657 mapwrite 0x5cab3d thru 0x5ce405 (0x38c9 bytes) 000395[1] 1766388585.420058 trunc from 0x5ce406 to 0x75be34 (0x18da2f bytes) 000396[0] 1766388585.440319 write 0x0e9705 thru 0x0f63f2 (0xccee bytes) 000397[1] 1766388585.473735 mapread 0x290e08 thru 0x2922f1 (0x14ea bytes) 000399[1] 1766388585.510620 trunc from 0x75be35 to 0x6afcd3 (0xac161 bytes) 000400[0] 1766388585.552359 read 0x12abb3 thru 0x12bbeb (0x1039 bytes) truncating to largest ever: 0x9e6e04 000401[1] 1766388585.566472 trunc from 0x6afcd4 to 0x9e6e03 (0x337130 bytes) 000404[0] 1766388585.660071 trunc from 0x9e6e04 to 0x0ed6f6 (0x8f970d bytes) 000405[1] 1766388585.735732 read 0x06c2ab thru 0x074a7e (0x87d4 bytes) 000407[0] 1766388585.801274 mapwrite 0x621ffe thru 0x6284dc (0x64df bytes) 000408[1] 1766388585.942412 trunc from 0x6284dd to 0x8f35e9 (0x2cb10d bytes) 000409[0] 1766388585.993189 mapwrite 0x1d3c49 thru 0x1e108a (0xd442 bytes) 000410[1] 1766388586.109425 mapread 0x47543f thru 0x476d7e (0x1940 bytes) 000412[0] 1766388586.171037 trunc from 0x8f35ea to 0x5ac20b (0x3473de bytes) 000415[1] 1766388586.212745 trunc from 0x5ac20c to 0x4e41e0 (0xc802b bytes) 000416[0] 1766388586.242344 trunc from 0x4e41e1 to 0x3c2cab (0x121535 bytes) 000417[1] 1766388586.296066 read 0x10426e thru 0x1043e8 (0x17b bytes) 000418[1] 1766388586.377131 mapwrite 0x275694 thru 0x27d901 (0x826e bytes) 000420[0] 1766388586.507288 mapread 0x1b8b7e thru 0x1c63cc (0xd84f bytes) 000424[0] 1766388586.561956 write 0x3f088f thru 0x3fbd13 (0xb485 bytes) 000428[0] 1766388586.596458 write 0x845364 thru 0x84fcde (0xa97b bytes) 000431[0] 1766388586.638322 write 0x91ad6e thru 0x92108c (0x631f bytes) 000433[0] 1766388586.639904 trunc from 0x92108d to 0x2ed734 (0x633958 bytes) 000434[1] 1766388586.673119 write 0x24e84d thru 0x256d12 (0x84c6 bytes) 000437[0] 1766388586.753933 read 0x0924d2 thru 0x09f2b0 (0xcddf bytes) 000440[0] 1766388586.805862 mapread 0x058f5b thru 0x05ab8d (0x1c33 bytes) 000442[1] 1766388586.837047 write 0x958a31 thru 0x95e3e6 (0x59b6 bytes) 000443[0] 1766388586.869014 write 0x4292db thru 0x43213e (0x8e64 bytes) 000445[0] 1766388586.903968 write 0x0a5277 thru 0x0ae676 (0x9400 bytes) 000446[0] 1766388586.994194 write 0x40a97d thru 0x41954b (0xebcf bytes) 000447[1] 1766388587.002573 mapread 0x17c305 thru 0x18ab7b (0xe877 bytes) 000448[1] 1766388587.121308 write 0x8436dd thru 0x85134a (0xdc6e bytes) 000449[1] 1766388587.145591 mapread 0x424e45 thru 0x42d91b (0x8ad7 bytes) 000450[1] 1766388587.169398 write 0x0a84ed thru 0x0b5aec (0xd600 bytes) 000451[1] 1766388587.190324 trunc from 0x95e3e7 to 0x4248ff (0x539ae7 bytes) 000452[0] 1766388587.224283 trunc from 0x424900 to 0x5b8a83 (0x194184 bytes) 000456[1] 1766388587.284718 mapwrite 0x3fe1b0 thru 0x405203 (0x7054 bytes) 000458[1] 1766388587.415944 trunc from 0x5b8a84 to 0x6e7d3b (0x12f2b8 bytes) 000460[1] 1766388587.458920 mapwrite 0x0701e5 thru 0x072098 (0x1eb4 bytes) 000461[1] 1766388587.509536 mapwrite 0x320048 thru 0x32b7eb (0xb7a4 bytes) 000462[1] 1766388587.595870 write 0x153a3c thru 0x15eef9 (0xb4be bytes) 000464[1] 1766388587.607039 mapread 0x53375a thru 0x53f444 (0xbceb bytes) 000465[1] 1766388587.657755 mapread 0x009aa6 thru 0x00cbc2 (0x311d bytes) 000470[1] 1766388587.795479 write 0x647682 thru 0x64fab5 (0x8434 bytes) 000473[0] 1766388587.818003 mapread 0x39dfe6 thru 0x3a4117 (0x6132 bytes) 000476[1] 1766388587.892725 trunc from 0x6e7d3c to 0x2f9567 (0x3ee7d4 bytes) 000479[0] 1766388587.977278 mapread 0x14f6ef thru 0x15a32f (0xac41 bytes) 000481[0] 1766388588.059106 read 0x06f707 thru 0x07a946 (0xb240 bytes) 000483[0] 1766388588.110237 mapread 0x1d8a87 thru 0x1dc17b (0x36f5 bytes) 000485[0] 1766388588.128841 read 0x23c36b thru 0x24b2ad (0xef43 bytes) 000486[0] 1766388588.144930 trunc from 0x2f9568 to 0x68050d (0x386fa6 bytes) 000487[1] 1766388588.180824 read 0x1e319c thru 0x1ee847 (0xb6ac bytes) 000488[0] 1766388588.254384 write 0x7404f8 thru 0x74df8f (0xda98 bytes) 000489[0] 1766388588.258713 write 0x91c20e thru 0x926aa4 (0xa897 bytes) 000490[1] 1766388588.281639 write 0x3e7146 thru 0x3f02fe (0x91b9 bytes) 000492[1] 1766388588.356471 write 0x9abc51 thru 0x9b3a3f (0x7def bytes) 000493[1] 1766388588.373793 write 0x74ce39 thru 0x75b7fd (0xe9c5 bytes) 000494[0] 1766388588.469433 mapread 0x57d42b thru 0x5879c7 (0xa59d bytes) 000496[0] 1766388588.558419 write 0x5b6792 thru 0x5c0dbd (0xa62c bytes) 000497[1] 1766388588.584312 read 0x375f7a thru 0x37c02d (0x60b4 bytes) 000499[0] 1766388588.621167 write 0x945549 thru 0x94b7f9 (0x62b1 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8588 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x914289 000001[1] 1766388589.239310 trunc from 00000000 to 0x914288 (0x914289 bytes) 000003[1] 1766388589.277245 mapwrite 0x0c7000 thru 0x0cfbd3 (0x8bd4 bytes) 000004[1] 1766388589.360325 write_OD 0x7bc000 thru 0x7c9fff (0xe000 bytes) 000005[1] 1766388589.397116 read_OD 0x898000 thru 0x89efff (0x7000 bytes) 000006[1] 1766388589.410246 mapwrite 0x6b8000 thru 0x6bf96a (0x796b bytes) 000007[0] 1766388589.479411 trunc from 0x914289 to 0x85590d (0xbe97b bytes) 000009[0] 1766388589.556938 trunc from 0x85590e to 0x1de084 (0x677889 bytes) 000010[0] 1766388589.586700 mapwrite 0x1ea000 thru 0x1ee488 (0x4489 bytes) 000011[0] 1766388589.681227 write 0x203000 thru 0x20ba24 (0x8a25 bytes) 000014[1] 1766388589.740794 trunc from 0x20ba25 to 0x57d167 (0x371743 bytes) 000017[1] 1766388589.803290 mapread 0x15f000 thru 0x16cb0b (0xdb0c bytes) 000020[0] 1766388589.881282 trunc from 0x57d168 to 0x091137 (0x4ec030 bytes) 000021[0] 1766388589.952882 write 0x89c000 thru 0x8abd5e (0xfd5f bytes) 000022[0] 1766388589.969152 mapread 0x5cc000 thru 0x5d8431 (0xc432 bytes) 000024[0] 1766388590.050761 mapwrite 0x2a3000 thru 0x2a74a9 (0x44aa bytes) 000025[1] 1766388590.134295 trunc from 0x8abd5f to 0x1a6cad (0x7050b1 bytes) 000030[1] 1766388590.224268 trunc from 0x1a6cae to 0x3ac98e (0x205ce1 bytes) 000033[1] 1766388590.292286 write_OD 0x42a000 thru 0x436fff (0xd000 bytes) 000034[1] 1766388590.343898 read_OD 0x10e000 thru 0x114fff (0x7000 bytes) 000037[1] 1766388590.354821 mapread 0x078000 thru 0x07a854 (0x2855 bytes) 000038[1] 1766388590.363731 trunc from 0x437000 to 0x12c518 (0x30aae7 bytes) 000039[0] 1766388590.387162 mapread 0x102000 thru 0x108ed2 (0x6ed3 bytes) 000040[0] 1766388590.420654 read 0x038000 thru 0x0461a9 (0xe1aa bytes) 000044[0] 1766388590.431024 read 0x015000 thru 0x017a70 (0x2a71 bytes) 000048[1] 1766388590.448963 mapread 0x07f000 thru 0x08db66 (0xeb67 bytes) 000050[0] 1766388590.464430 trunc from 0x12c519 to 0x8ca513 (0x79dffb bytes) 000054[0] 1766388590.532101 mapread 0x7e4000 thru 0x7e7b83 (0x3b84 bytes) 000055[1] 1766388590.612048 write_OD 0x872000 thru 0x876fff (0x5000 bytes) 000057[1] 1766388590.652414 trunc from 0x8ca514 to 0x1c055f (0x709fb4 bytes) 000058[1] 1766388590.674235 mapread 0x122000 thru 0x13180a (0xf80b bytes) 000059[0] 1766388590.708450 read 0x0f5000 thru 0x0fec44 (0x9c45 bytes) 000060[1] 1766388590.779247 mapread 0x0b3000 thru 0x0b819e (0x519f bytes) 000061[0] 1766388590.789810 mapread 0x0c3000 thru 0x0cd69c (0xa69d bytes) 000063[1] 1766388590.791891 read_OD 0x0e5000 thru 0x0effff (0xb000 bytes) 000064[1] 1766388590.800628 read_OD 0x120000 thru 0x125fff (0x6000 bytes) 000065[1] 1766388590.807665 trunc from 0x1c0560 to 0x36d6f0 (0x1ad191 bytes) 000066[1] 1766388590.847991 mapwrite 0x65e000 thru 0x66b3f9 (0xd3fa bytes) 000067[0] 1766388590.969356 mapwrite 0x789000 thru 0x795209 (0xc20a bytes) 000068[0] 1766388591.135875 mapread 0x5e7000 thru 0x5eed38 (0x7d39 bytes) 000069[1] 1766388591.160301 mapread 0x772000 thru 0x781e52 (0xfe53 bytes) 000070[0] 1766388591.203225 trunc from 0x79520a to 0x880f7d (0xebd74 bytes) 000071[0] 1766388591.245797 trunc from 0x880f7e to 0x86a939 (0x16644 bytes) 000072[0] 1766388591.264279 mapwrite 0x220000 thru 0x224b3f (0x4b40 bytes) 000077[1] 1766388591.414084 write_OD 0x18b000 thru 0x18efff (0x4000 bytes) 000078[0] 1766388591.463305 write 0x398000 thru 0x3a797f (0xf980 bytes) 000079[1] 1766388591.477849 mapwrite 0x9fe000 thru 0x9fe380 (0x381 bytes) 000081[0] 1766388591.680457 write 0x493000 thru 0x499c38 (0x6c39 bytes) 000082[1] 1766388591.702514 mapwrite 0x66d000 thru 0x66fc79 (0x2c7a bytes) 000086[1] 1766388591.844268 read_OD 0x9a1000 thru 0x9a5fff (0x5000 bytes) 000088[1] 1766388591.863700 write_OD 0x6f2000 thru 0x6fcfff (0xb000 bytes) 000089[0] 1766388591.921453 trunc from 0x9fe381 to 0x87f2f0 (0x17f090 bytes) 000090[0] 1766388591.952030 mapwrite 0x9f5000 thru 0x9fa5bd (0x55be bytes) 000091[1] 1766388592.037668 mapwrite 0x984000 thru 0x98cd86 (0x8d87 bytes) 000093[1] 1766388592.118032 write_OD 0x3b4000 thru 0x3bcfff (0x9000 bytes) 000094[0] 1766388592.175834 mapwrite 0x5b1000 thru 0x5b884e (0x784f bytes) 000097[1] 1766388592.269315 trunc from 0x9fa5be to 0x24bd2a (0x7ae893 bytes) 000098[0] 1766388592.293058 trunc from 0x24bd2b to 0x272f93 (0x27269 bytes) 000106[0] 1766388592.399141 mapwrite 0x12c000 thru 0x131a88 (0x5a89 bytes) 000109[1] 1766388592.577612 mapwrite 0x4b8000 thru 0x4b90c8 (0x10c9 bytes) 000112[0] 1766388592.696105 mapread 0x285000 thru 0x28793d (0x293e bytes) 000121[0] 1766388592.727665 trunc from 0x4b90c9 to 0x14a920 (0x36e7a8 bytes) 000122[0] 1766388592.753963 read 0x007000 thru 0x016860 (0xf861 bytes) 000124[0] 1766388592.799655 read 0x109000 thru 0x10f74c (0x674d bytes) 000125[0] 1766388592.807788 read 0x034000 thru 0x037097 (0x3098 bytes) 000126[0] 1766388592.808320 read 0x0a6000 thru 0x0aa636 (0x4637 bytes) 000127[1] 1766388592.808711 trunc from 0x14a921 to 0x639b32 (0x4ef212 bytes) 000128[1] 1766388592.838785 trunc from 0x639b33 to 0x14571e (0x4f4414 bytes) 000130[0] 1766388592.880147 read 0x059000 thru 0x06504f (0xc050 bytes) 000131[0] 1766388592.913729 write 0x110000 thru 0x11afeb (0xafec bytes) 000132[0] 1766388592.934071 trunc from 0x14571f to 0x3d294e (0x28d230 bytes) 000133[0] 1766388592.950403 mapwrite 0x883000 thru 0x88c868 (0x9869 bytes) 000134[0] 1766388593.124587 read 0x721000 thru 0x727033 (0x6034 bytes) truncating to largest ever: 0x92d547 000135[1] 1766388593.125561 trunc from 0x88c869 to 0x92d546 (0xa0cde bytes) 000137[1] 1766388593.173547 read_OD 0x098000 thru 0x09afff (0x3000 bytes) 000139[0] 1766388593.215857 read 0x106000 thru 0x106dfd (0xdfe bytes) 000142[0] 1766388593.222334 mapwrite 0x199000 thru 0x19cfde (0x3fdf bytes) 000147[0] 1766388593.341924 write 0x7e3000 thru 0x7edf2d (0xaf2e bytes) 000149[0] 1766388593.364482 mapwrite 0x00c000 thru 0x016c67 (0xac68 bytes) 000150[1] 1766388593.475741 mapwrite 0x98c000 thru 0x98fb08 (0x3b09 bytes) 000156[0] 1766388593.609266 mapwrite 0x56e000 thru 0x5747bb (0x67bc bytes) 000157[0] 1766388593.683902 mapread 0x30b000 thru 0x30e16e (0x316f bytes) 000158[0] 1766388593.712342 trunc from 0x98fb09 to 0x087689 (0x90847f bytes) 000159[1] 1766388593.735786 read_OD 0x083000 thru 0x086fff (0x4000 bytes) 000162[0] 1766388593.755656 mapwrite 0x316000 thru 0x31a592 (0x4593 bytes) 000164[0] 1766388593.831732 read 0x151000 thru 0x15e4e6 (0xd4e7 bytes) 000165[1] 1766388593.879157 write_OD 0x20e000 thru 0x20efff (0x1000 bytes) 000169[1] 1766388593.945523 read_OD 0x1c5000 thru 0x1c5fff (0x1000 bytes) 000170[0] 1766388593.949910 read 0x312000 thru 0x316e99 (0x4e9a bytes) 000171[0] 1766388594.002431 read 0x06d000 thru 0x0712b5 (0x42b6 bytes) 000172[1] 1766388594.004060 mapread 0x089000 thru 0x091019 (0x801a bytes) 000173[1] 1766388594.052334 read_OD 0x237000 thru 0x23dfff (0x7000 bytes) 000174[1] 1766388594.064765 mapread 0x061000 thru 0x06a417 (0x9418 bytes) 000177[1] 1766388594.066588 read_OD 0x204000 thru 0x20bfff (0x8000 bytes) 000181[0] 1766388594.076945 read 0x2c1000 thru 0x2c7ae9 (0x6aea bytes) 000187[1] 1766388594.078319 read_OD 0x1ae000 thru 0x1affff (0x2000 bytes) 000188[0] 1766388594.084483 trunc from 0x31a593 to 0x09b316 (0x27f27c bytes) 000190[0] 1766388594.131444 mapwrite 0x2ad000 thru 0x2b09c0 (0x39c1 bytes) 000192[0] 1766388594.259427 mapread 0x03f000 thru 0x041fd0 (0x2fd1 bytes) 000195[1] 1766388594.302090 read_OD 0x0ec000 thru 0x0f6fff (0xb000 bytes) 000197[1] 1766388594.327777 mapread 0x129000 thru 0x130d74 (0x7d75 bytes) 000198[1] 1766388594.386709 mapwrite 0x881000 thru 0x88b4bd (0xa4be bytes) 000200[0] 1766388594.553804 mapread 0x08f000 thru 0x099119 (0xa11a bytes) 000202[1] 1766388594.661984 trunc from 0x88b4be to 0x075110 (0x8163ad bytes) 000203[0] 1766388594.695676 mapread 0x012000 thru 0x01580c (0x380d bytes) 000204[0] 1766388594.739018 trunc from 0x075111 to 0x7f04b8 (0x77b3a8 bytes) 000205[1] 1766388594.772076 trunc from 0x7f04b9 to 0x8da3cf (0xe9f17 bytes) 000206[1] 1766388594.800250 mapread 0x1fc000 thru 0x205816 (0x9817 bytes) 000208[0] 1766388594.895233 mapread 0x057000 thru 0x0609db (0x99dc bytes) 000212[1] 1766388594.966304 write_OD 0x00b000 thru 0x015fff (0xb000 bytes) 000213[0] 1766388595.007936 write 0x6d0000 thru 0x6daceb (0xacec bytes) 000219[1] 1766388595.079145 write_OD 0x783000 thru 0x789fff (0x7000 bytes) 000220[1] 1766388595.143071 trunc from 0x8da3d0 to 0x2f02b2 (0x5ea11d bytes) 000221[1] 1766388595.202005 mapwrite 0x09e000 thru 0x0acfd1 (0xefd2 bytes) 000222[0] 1766388595.374582 write 0x465000 thru 0x470bb8 (0xbbb9 bytes) 000223[0] 1766388595.424799 write 0x8c9000 thru 0x8d0634 (0x7635 bytes) 000226[0] 1766388595.445323 mapread 0x3c3000 thru 0x3c5f93 (0x2f94 bytes) 000227[0] 1766388595.476307 write 0x63b000 thru 0x648b99 (0xdb9a bytes) 000228[0] 1766388595.507237 trunc from 0x8d0635 to 0x80584e (0xcade6 bytes) 000230[0] 1766388595.558399 write 0x16e000 thru 0x178393 (0xa394 bytes) 000231[0] 1766388595.593245 read 0x60d000 thru 0x6152d1 (0x82d2 bytes) 000234[0] 1766388595.628836 mapwrite 0x29f000 thru 0x2a336e (0x436f bytes) 000236[1] 1766388595.724623 trunc from 0x80584f to 0x03b632 (0x7ca21c bytes) 000237[0] 1766388595.763258 mapread 0x02a000 thru 0x02ffd7 (0x5fd8 bytes) 000239[0] 1766388595.805028 mapread 0x009000 thru 0x015972 (0xc973 bytes) 000241[0] 1766388595.824463 trunc from 0x03b633 to 0x33a88f (0x2ff25d bytes) 000242[1] 1766388595.880954 write_OD 0x035000 thru 0x037fff (0x3000 bytes) 000243[1] 1766388595.924749 write_OD 0x9a3000 thru 0x9acfff (0xa000 bytes) 000245[1] 1766388595.973671 read_OD 0x157000 thru 0x162fff (0xc000 bytes) 000247[0] 1766388595.985956 write 0x1cd000 thru 0x1d651a (0x951b bytes) 000248[1] 1766388595.995471 mapwrite 0x31e000 thru 0x325772 (0x7773 bytes) 000249[1] 1766388596.085134 read_OD 0x337000 thru 0x345fff (0xf000 bytes) 000255[1] 1766388596.101356 mapread 0x010000 thru 0x01ea34 (0xea35 bytes) 000262[0] 1766388596.122309 mapread 0x099000 thru 0x0a8afa (0xfafb bytes) 000264[1] 1766388596.139590 trunc from 0x9ad000 to 0x326e8e (0x686171 bytes) 000268[0] 1766388596.180030 read 0x18d000 thru 0x18df67 (0xf68 bytes) 000269[0] 1766388596.236885 read 0x315000 thru 0x315666 (0x667 bytes) 000271[1] 1766388596.237613 mapwrite 0x6d3000 thru 0x6d9c06 (0x6c07 bytes) 000272[1] 1766388596.357866 read_OD 0x1e2000 thru 0x1e3fff (0x2000 bytes) 000276[0] 1766388596.384773 write 0x5b5000 thru 0x5bd11a (0x811b bytes) 000277[1] 1766388596.450362 trunc from 0x6d9c07 to 0x1ebb58 (0x4ee0ae bytes) 000278[0] 1766388596.504340 mapwrite 0x6a0000 thru 0x6a5e82 (0x5e83 bytes) 000279[0] 1766388596.600968 trunc from 0x6a5e83 to 0x24ae37 (0x45b04b bytes) 000280[1] 1766388596.645463 mapread 0x0cd000 thru 0x0d50ef (0x80f0 bytes) 000282[1] 1766388596.682223 read_OD 0x081000 thru 0x083fff (0x3000 bytes) 000285[0] 1766388596.687610 read 0x171000 thru 0x17652d (0x552e bytes) 000287[1] 1766388596.724814 write_OD 0x949000 thru 0x955fff (0xd000 bytes) 000288[0] 1766388596.769904 mapread 0x620000 thru 0x621a25 (0x1a26 bytes) 000291[1] 1766388596.817324 mapwrite 0x6e4000 thru 0x6eb8e7 (0x78e8 bytes) 000292[1] 1766388596.996393 write_OD 0x2d8000 thru 0x2e1fff (0xa000 bytes) 000293[1] 1766388597.060527 trunc from 0x956000 to 0x3c1386 (0x594c79 bytes) 000294[1] 1766388597.093482 mapread 0x115000 thru 0x121851 (0xc852 bytes) 000295[0] 1766388597.190947 mapwrite 0x120000 thru 0x12c81e (0xc81f bytes) 000296[0] 1766388597.367270 mapwrite 0x62f000 thru 0x63b052 (0xc053 bytes) 000299[0] 1766388597.473221 write 0x9b3000 thru 0x9b4d5a (0x1d5b bytes) 000301[1] 1766388597.507790 mapread 0x68c000 thru 0x6985a8 (0xc5a9 bytes) truncating to largest ever: 0x954f36 000302[1] 1766388597.609571 trunc from 0x9b4d5b to 0x954f35 (0x5fe25 bytes) 000306[0] 1766388597.694794 mapwrite 0x2dd000 thru 0x2e2838 (0x5839 bytes) 000311[1] 1766388597.850738 trunc from 0x954f36 to 0x1c181d (0x793718 bytes) 000312[1] 1766388597.887977 write_OD 0x62c000 thru 0x638fff (0xd000 bytes) 000313[0] 1766388597.936588 mapwrite 0x6b0000 thru 0x6ba645 (0xa646 bytes) 000314[1] 1766388598.055371 write_OD 0x643000 thru 0x644fff (0x2000 bytes) 000315[0] 1766388598.095899 trunc from 0x6ba646 to 0x6d48f5 (0x1a2b0 bytes) 000316[1] 1766388598.111657 trunc from 0x6d48f6 to 0x4d076b (0x20418a bytes) 000317[0] 1766388598.136459 read 0x137000 thru 0x13ce78 (0x5e79 bytes) 000318[0] 1766388598.157471 mapread 0x440000 thru 0x44f1c9 (0xf1ca bytes) 000320[1] 1766388598.192696 mapread 0x086000 thru 0x088d44 (0x2d45 bytes) 000322[0] 1766388598.211299 write 0x160000 thru 0x165152 (0x5153 bytes) 000327[0] 1766388598.277252 mapwrite 0x065000 thru 0x06a3e8 (0x53e9 bytes) 000328[0] 1766388598.338684 mapread 0x3cf000 thru 0x3dabf4 (0xbbf5 bytes) 000329[0] 1766388598.363975 read 0x378000 thru 0x380761 (0x8762 bytes) 000330[1] 1766388598.375023 trunc from 0x4d076c to 0x751429 (0x280cbe bytes) 000335[1] 1766388598.436327 trunc from 0x75142a to 0x6194c2 (0x137f67 bytes) 000337[0] 1766388598.481822 read 0x0ac000 thru 0x0b5e38 (0x9e39 bytes) 000338[1] 1766388598.495205 mapread 0x07a000 thru 0x083c44 (0x9c45 bytes) 000340[0] 1766388598.536977 write 0x716000 thru 0x71b9cd (0x59ce bytes) 000343[1] 1766388598.576962 read_OD 0x170000 thru 0x177fff (0x8000 bytes) 000345[1] 1766388598.601669 mapwrite 0x42d000 thru 0x434d21 (0x7d22 bytes) 000346[1] 1766388598.660748 trunc from 0x71b9ce to 0x4b2c95 (0x268d38 bytes) 000350[0] 1766388598.690595 write 0x954000 thru 0x95fc1e (0xbc1f bytes) 000352[0] 1766388598.718367 write 0x05d000 thru 0x06471c (0x771d bytes) 000354[0] 1766388598.729361 mapwrite 0x8cc000 thru 0x8d0c1e (0x4c1f bytes) 000356[0] 1766388598.800347 write 0x1ce000 thru 0x1d86b5 (0xa6b6 bytes) 000358[1] 1766388598.808804 mapwrite 0x1e1000 thru 0x1ecb06 (0xbb07 bytes) 000359[1] 1766388598.918393 write_OD 0x06e000 thru 0x075fff (0x8000 bytes) 000360[0] 1766388598.955724 mapwrite 0x56b000 thru 0x576395 (0xb396 bytes) 000361[0] 1766388599.032031 write 0x164000 thru 0x16a9c1 (0x69c2 bytes) 000363[0] 1766388599.075192 read 0x68c000 thru 0x68f377 (0x3378 bytes) 000364[0] 1766388599.080446 mapwrite 0x78f000 thru 0x79e4eb (0xf4ec bytes) 000372[0] 1766388599.225820 trunc from 0x95fc1f to 0x23c914 (0x72330a bytes) 000373[0] 1766388599.239684 mapwrite 0x81b000 thru 0x81fa7a (0x4a7b bytes) 000375[1] 1766388599.327352 write_OD 0x3c6000 thru 0x3cdfff (0x8000 bytes) 000377[1] 1766388599.373683 read_OD 0x543000 thru 0x550fff (0xe000 bytes) 000380[0] 1766388599.384734 read 0x5b9000 thru 0x5c1cb6 (0x8cb7 bytes) 000381[0] 1766388599.398381 mapwrite 0x242000 thru 0x244401 (0x2402 bytes) 000384[0] 1766388599.483603 mapwrite 0x413000 thru 0x4220f7 (0xf0f8 bytes) 000385[0] 1766388599.553565 read 0x5d2000 thru 0x5dc8ae (0xa8af bytes) 000387[0] 1766388599.584903 mapwrite 0x101000 thru 0x108b88 (0x7b89 bytes) 000388[1] 1766388599.634850 trunc from 0x81fa7b to 0x4527f3 (0x3cd287 bytes) 000393[0] 1766388599.672592 read 0x04a000 thru 0x04e388 (0x4389 bytes) 000394[0] 1766388599.685428 write 0x2d5000 thru 0x2d719e (0x219f bytes) 000395[1] 1766388599.701449 write_OD 0x81c000 thru 0x82afff (0xf000 bytes) 000396[1] 1766388599.740782 mapwrite 0x6ea000 thru 0x6f2233 (0x8234 bytes) 000398[0] 1766388599.802607 read 0x2a2000 thru 0x2a9435 (0x7436 bytes) 000399[0] 1766388599.813938 mapread 0x258000 thru 0x25c515 (0x4516 bytes) 000400[1] 1766388599.825261 mapread 0x501000 thru 0x504979 (0x397a bytes) 000403[1] 1766388599.839575 trunc from 0x82b000 to 0x5ecfa1 (0x23e05e bytes) 000404[1] 1766388599.859058 mapread 0x5d1000 thru 0x5dd7ec (0xc7ed bytes) 000408[0] 1766388599.929923 mapwrite 0x772000 thru 0x7754b7 (0x34b8 bytes) 000409[0] 1766388599.989853 read 0x4af000 thru 0x4b3e0a (0x4e0b bytes) 000410[0] 1766388599.996115 mapread 0x326000 thru 0x32beb8 (0x5eb9 bytes) 000411[1] 1766388600.016296 trunc from 0x7754b8 to 0x6c219f (0xb3318 bytes) 000414[0] 1766388600.040760 read 0x64e000 thru 0x65a414 (0xc415 bytes) 000417[1] 1766388600.061931 mapwrite 0x7a7000 thru 0x7b2fe9 (0xbfea bytes) 000418[0] 1766388600.154689 read 0x120000 thru 0x12fcba (0xfcbb bytes) 000424[1] 1766388600.198011 write_OD 0x1b5000 thru 0x1befff (0xa000 bytes) 000426[1] 1766388600.247852 write_OD 0x439000 thru 0x43dfff (0x5000 bytes) 000429[1] 1766388600.292643 trunc from 0x7b2fea to 0x7a351e (0xfacb bytes) 000431[0] 1766388600.337079 trunc from 0x7a351f to 0x479a3d (0x329ae1 bytes) 000432[1] 1766388600.365042 read_OD 0x053000 thru 0x053fff (0x1000 bytes) 000435[0] 1766388600.392972 trunc from 0x479a3e to 0x6c180f (0x247dd2 bytes) 000436[1] 1766388600.442995 read_OD 0x019000 thru 0x026fff (0xe000 bytes) 000438[1] 1766388600.475885 write_OD 0x2a7000 thru 0x2aafff (0x4000 bytes) 000439[0] 1766388600.519063 trunc from 0x6c1810 to 0x2f1730 (0x3d00df bytes) 000441[0] 1766388600.542611 read 0x04e000 thru 0x056147 (0x8148 bytes) 000444[0] 1766388600.571259 write 0x9a1000 thru 0x9a5344 (0x4345 bytes) 000449[0] 1766388600.589121 mapread 0x807000 thru 0x8138d5 (0xc8d6 bytes) 000454[1] 1766388600.705322 write_OD 0x679000 thru 0x687fff (0xf000 bytes) 000455[1] 1766388600.775877 write_OD 0x740000 thru 0x74dfff (0xe000 bytes) 000456[0] 1766388600.829557 read 0x867000 thru 0x874505 (0xd506 bytes) 000459[1] 1766388600.843992 mapwrite 0x820000 thru 0x8220e0 (0x20e1 bytes) 000460[0] 1766388600.935166 read 0x6f6000 thru 0x70183d (0xb83e bytes) 000461[0] 1766388600.956422 write 0x8cf000 thru 0x8d8403 (0x9404 bytes) 000462[0] 1766388600.976248 mapread 0x2c6000 thru 0x2cca14 (0x6a15 bytes) 000466[0] 1766388601.039241 read 0x704000 thru 0x711ea2 (0xdea3 bytes) 000468[0] 1766388601.090160 mapwrite 0x272000 thru 0x27ab09 (0x8b0a bytes) 000470[1] 1766388601.169694 trunc from 0x9a5345 to 0x9450e8 (0x6025c bytes) 000475[0] 1766388601.234937 write 0x91f000 thru 0x922648 (0x3649 bytes) 000477[0] 1766388601.251026 trunc from 0x9450e9 to 0x3e8c7b (0x55c46d bytes) 000478[0] 1766388601.288217 mapread 0x01e000 thru 0x029c36 (0xbc37 bytes) 000479[0] 1766388601.326225 write 0x7f7000 thru 0x804889 (0xd88a bytes) 000480[0] 1766388601.344209 trunc from 0x80488a to 0x5ec3bb (0x2184ce bytes) 000481[1] 1766388601.368874 trunc from 0x5ec3bc to 0x43382c (0x1b8b8f bytes) 000482[1] 1766388601.392127 read_OD 0x0d5000 thru 0x0e0fff (0xc000 bytes) 000484[0] 1766388601.409452 write 0x6ab000 thru 0x6ad9f6 (0x29f7 bytes) 000485[0] 1766388601.421995 read 0x220000 thru 0x22cb3c (0xcb3d bytes) 000486[1] 1766388601.429297 read_OD 0x5fc000 thru 0x606fff (0xb000 bytes) 000489[1] 1766388601.461369 mapwrite 0x8ee000 thru 0x8fa027 (0xc028 bytes) 000490[0] 1766388601.559844 trunc from 0x8fa028 to 0x785f63 (0x1740c4 bytes) 000491[1] 1766388601.605380 trunc from 0x785f64 to 0x1316e5 (0x65487e bytes) 000492[0] 1766388601.643948 write 0x637000 thru 0x63db90 (0x6b91 bytes) 000493[1] 1766388601.674574 trunc from 0x63db91 to 0x785361 (0x1477d1 bytes) 000494[0] 1766388601.707924 mapread 0x5e9000 thru 0x5eb702 (0x2703 bytes) 000498[0] 1766388601.740166 read 0x754000 thru 0x757781 (0x3782 bytes) All operations completed A-OK! PASS 16a (36s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:30:04 (1766388604) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.222327 s, 37.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.163445 s, 51.3 MB/s Chance of close/open is 1 in 50 Seed set to 8606 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x847247 000010[1] 1766388606.564386 trunc from 00000000 to 0x847246 (0x847247 bytes) 000013[0] 1766388606.642568 read 0x6c0d0c thru 0x6c2b21 (0x1e16 bytes) 000014[0] 1766388606.662975 mapread 0x1ff734 thru 0x1ffd6d (0x63a bytes) 000016[0] 1766388606.690855 trunc from 0x847247 to 0x1f7d21 (0x64f525 bytes) 000017[0] 1766388606.732013 mapwrite 0x523dad thru 0x525b35 (0x1d89 bytes) 000021[0] 1766388606.845248 mapread 0x0ff5d7 thru 0x10140f (0x1e39 bytes) 000022[1] 1766388606.874152 mapread 0x1dacb8 thru 0x1dc48b (0x17d4 bytes) 000024[0] 1766388606.971249 mapwrite 0x59daaa thru 0x59e807 (0xd5e bytes) 000026[0] 1766388607.136419 mapread 0x08570b thru 0x0857ff (0xf5 bytes) 000027[1] 1766388607.143905 mapwrite 0x466120 thru 0x466596 (0x477 bytes) 000028[0] 1766388607.246978 mapwrite 0x470d95 thru 0x4724de (0x174a bytes) 000031[0] 1766388607.370987 write 0x216866 thru 0x216a4a (0x1e5 bytes) 000032[1] 1766388607.395685 read 0x08d923 thru 0x08de2b (0x509 bytes) 000034[0] 1766388607.434967 trunc from 0x59e808 to 0x64a602 (0xabdfb bytes) 000035[1] 1766388607.466338 mapread 0x599600 thru 0x59a802 (0x1203 bytes) 000036[0] 1766388607.508427 read 0x0b7080 thru 0x0b7be7 (0xb68 bytes) 000038[0] 1766388607.529149 write 0x3bdf4e thru 0x3be148 (0x1fb bytes) 000039[0] 1766388607.548482 mapwrite 0x83a3bf thru 0x83b101 (0xd43 bytes) 000042[0] 1766388607.668144 read 0x66a894 thru 0x66c4a4 (0x1c11 bytes) 000043[0] 1766388607.702015 read 0x5d5801 thru 0x5d7765 (0x1f65 bytes) 000044[0] 1766388607.720122 write 0x319c7c thru 0x31aecd (0x1252 bytes) truncating to largest ever: 0x956c7c 000048[1] 1766388607.769893 trunc from 0x83b102 to 0x956c7b (0x11bb7a bytes) 000053[0] 1766388607.921076 mapread 0x1f27b8 thru 0x1f30cd (0x916 bytes) 000055[0] 1766388607.967377 write 0x30ea58 thru 0x31065e (0x1c07 bytes) 000060[1] 1766388608.076893 trunc from 0x956c7c to 0x8f9424 (0x5d857 bytes) 000062[1] 1766388608.135788 read 0x6da2d3 thru 0x6da32a (0x58 bytes) 000063[0] 1766388608.171433 write 0x8a252d thru 0x8a4346 (0x1e1a bytes) 000069[0] 1766388608.283134 trunc from 0x8f9425 to 0x1b3d39 (0x7456eb bytes) 000071[1] 1766388608.319830 trunc from 0x1b3d3a to 0x645d95 (0x49205c bytes) 000072[0] 1766388608.395788 mapwrite 0x996d22 thru 0x996da0 (0x7f bytes) 000074[0] 1766388608.488111 mapwrite 0x14ac25 thru 0x14c465 (0x1841 bytes) 000077[0] 1766388608.586991 write 0x523768 thru 0x524b64 (0x13fd bytes) 000078[1] 1766388608.606677 mapread 0x50c152 thru 0x50c270 (0x11f bytes) 000081[0] 1766388608.663965 write 0x17bd98 thru 0x17c14e (0x3b7 bytes) 000083[1] 1766388608.665198 read 0x276cf1 thru 0x2770ed (0x3fd bytes) 000085[1] 1766388608.690794 mapwrite 0x8201aa thru 0x8213d2 (0x1229 bytes) 000086[1] 1766388608.779127 mapwrite 0x5c9b4f thru 0x5cb72b (0x1bdd bytes) 000089[1] 1766388608.875505 mapread 0x6ff617 thru 0x7004e3 (0xecd bytes) 000091[1] 1766388608.940204 mapwrite 0x72f25f thru 0x72ff2c (0xcce bytes) 000092[0] 1766388609.015749 read 0x789975 thru 0x78a8d3 (0xf5f bytes) 000093[0] 1766388609.036229 trunc from 0x996da1 to 0x19bced (0x7fb0b3 bytes) 000094[0] 1766388609.097810 trunc from 0x19bcee to 0x8bb5dc (0x71f8ef bytes) 000097[1] 1766388609.149158 write 0x07ec9e thru 0x080b60 (0x1ec3 bytes) 000099[0] 1766388609.223943 trunc from 0x8bb5dd to 0x8095aa (0xb2032 bytes) 000101[1] 1766388609.277553 mapwrite 0x4de967 thru 0x4deba4 (0x23e bytes) 000102[1] 1766388609.348119 mapread 0x2f33ae thru 0x2f4ef5 (0x1b48 bytes) 000104[0] 1766388609.376456 trunc from 0x8095ab to 0x93afc1 (0x131a17 bytes) truncating to largest ever: 0x98e7df 000107[0] 1766388609.449868 trunc from 0x93afc2 to 0x98e7de (0x5381d bytes) 000108[1] 1766388609.493983 mapwrite 0x7748e6 thru 0x774e1d (0x538 bytes) 000109[1] 1766388609.594953 trunc from 0x98e7df to 0x438671 (0x55616d bytes) 000113[1] 1766388609.686845 write 0x40701d thru 0x4080f3 (0x10d7 bytes) 000115[0] 1766388609.738016 trunc from 0x438672 to 0x1d0e81 (0x2677f0 bytes) 000117[0] 1766388609.779239 read 0x130bca thru 0x13229b (0x16d2 bytes) 000119[0] 1766388609.837433 write 0x210862 thru 0x2123e3 (0x1b82 bytes) 000120[1] 1766388609.838532 read 0x131e81 thru 0x13256a (0x6ea bytes) 000123[0] 1766388609.941777 read 0x17fa78 thru 0x18177f (0x1d08 bytes) 000126[0] 1766388609.978196 mapread 0x1d6265 thru 0x1d7425 (0x11c1 bytes) 000127[1] 1766388609.994174 write 0x3b9248 thru 0x3ba444 (0x11fd bytes) 000133[1] 1766388610.105627 read 0x01701f thru 0x017734 (0x716 bytes) 000134[1] 1766388610.183105 mapwrite 0x52d806 thru 0x52f0c9 (0x18c4 bytes) 000137[1] 1766388610.346479 write 0x83d444 thru 0x83eafa (0x16b7 bytes) 000140[0] 1766388610.385410 read 0x051ebc thru 0x052c5a (0xd9f bytes) 000146[0] 1766388610.486751 mapwrite 0x4c765c thru 0x4c7e40 (0x7e5 bytes) 000148[1] 1766388610.578121 trunc from 0x83eafb to 0x47cc48 (0x3c1eb2 bytes) 000150[1] 1766388610.641992 read 0x12ec57 thru 0x12f8ae (0xc58 bytes) 000152[0] 1766388610.648906 mapread 0x3f6d4c thru 0x3f7d58 (0x100d bytes) 000154[0] 1766388610.718616 write 0x0b60b5 thru 0x0b7cb3 (0x1bff bytes) 000157[0] 1766388610.791781 mapwrite 0x564cb2 thru 0x565a29 (0xd78 bytes) 000158[1] 1766388610.864922 mapread 0x279013 thru 0x27acd2 (0x1cc0 bytes) 000160[1] 1766388610.893445 write 0x0762cd thru 0x076ff7 (0xd2b bytes) 000163[1] 1766388610.957094 mapread 0x03d93a thru 0x03e077 (0x73e bytes) 000165[1] 1766388611.011532 trunc from 0x565a2a to 0x668a6d (0x103044 bytes) 000166[1] 1766388611.043049 mapwrite 0x83c9d3 thru 0x83e741 (0x1d6f bytes) 000167[1] 1766388611.129009 write 0x2779d4 thru 0x278712 (0xd3f bytes) 000171[0] 1766388611.165612 mapread 0x6fffaf thru 0x701c7c (0x1cce bytes) 000172[0] 1766388611.212091 mapwrite 0x1b1095 thru 0x1b26dd (0x1649 bytes) 000173[1] 1766388611.263372 read 0x628b33 thru 0x62aa05 (0x1ed3 bytes) 000174[1] 1766388611.270612 mapwrite 0x7327d4 thru 0x7347c1 (0x1fee bytes) 000175[0] 1766388611.344900 read 0x692667 thru 0x6928d0 (0x26a bytes) 000180[0] 1766388611.398586 mapread 0x500de5 thru 0x502143 (0x135f bytes) 000182[1] 1766388611.426031 read 0x8219ac thru 0x821bb9 (0x20e bytes) 000183[0] 1766388611.455692 write 0x80efcf thru 0x80f63b (0x66d bytes) 000184[1] 1766388611.486429 mapwrite 0x5c1478 thru 0x5c22c7 (0xe50 bytes) 000185[1] 1766388611.578602 mapread 0x472728 thru 0x473c58 (0x1531 bytes) 000186[0] 1766388611.626339 trunc from 0x83e742 to 0x5a619c (0x2985a5 bytes) 000187[1] 1766388611.654877 mapwrite 0x063109 thru 0x064f06 (0x1dfe bytes) 000188[1] 1766388611.742444 read 0x0297f8 thru 0x02adcc (0x15d5 bytes) 000190[0] 1766388611.768627 read 0x4a7053 thru 0x4a82d8 (0x1286 bytes) 000192[1] 1766388611.801154 mapwrite 0x8141e5 thru 0x81601d (0x1e39 bytes) 000193[0] 1766388611.912319 write 0x745836 thru 0x745d12 (0x4dd bytes) 000195[0] 1766388611.961616 trunc from 0x81601e to 0x523719 (0x2f2904 bytes) 000196[1] 1766388612.007714 read 0x396413 thru 0x397c7e (0x186c bytes) 000200[0] 1766388612.062615 mapwrite 0x3c397c thru 0x3c53ac (0x1a31 bytes) 000201[0] 1766388612.179755 mapread 0x30ce95 thru 0x30e66d (0x17d9 bytes) 000202[0] 1766388612.187572 mapread 0x4b5030 thru 0x4b6cb4 (0x1c85 bytes) 000206[0] 1766388612.264134 read 0x3039b4 thru 0x3046ff (0xd4c bytes) 000208[1] 1766388612.288683 trunc from 0x52371a to 0x67a73a (0x157021 bytes) 000209[0] 1766388612.317188 write 0x878b37 thru 0x879bde (0x10a8 bytes) 000210[1] 1766388612.340736 trunc from 0x879bdf to 0x1b843d (0x6c17a1 bytes) 000211[0] 1766388612.400743 mapread 0x030f29 thru 0x0323aa (0x1482 bytes) 000213[0] 1766388612.475361 read 0x06e6d8 thru 0x06f339 (0xc62 bytes) 000218[0] 1766388612.569130 trunc from 0x1b843e to 0x3bf6a3 (0x207266 bytes) 000221[1] 1766388612.638346 write 0x599b98 thru 0x59ab38 (0xfa1 bytes) 000223[0] 1766388612.678576 mapread 0x0cb378 thru 0x0cb972 (0x5fb bytes) 000224[0] 1766388612.736036 mapread 0x33ebe2 thru 0x340b76 (0x1f95 bytes) truncating to largest ever: 0x9f018e 000225[1] 1766388612.763111 trunc from 0x59ab39 to 0x9f018d (0x455655 bytes) 000226[0] 1766388612.804263 mapwrite 0x947fc0 thru 0x9486c3 (0x704 bytes) 000228[0] 1766388612.925580 write 0x84071e thru 0x841e9e (0x1781 bytes) 000230[0] 1766388612.965579 read 0x41c3b6 thru 0x41c5af (0x1fa bytes) 000231[1] 1766388612.985759 mapread 0x702f68 thru 0x704590 (0x1629 bytes) 000234[0] 1766388613.007736 mapwrite 0x2694d4 thru 0x26ad64 (0x1891 bytes) 000235[0] 1766388613.047887 write 0x06468f thru 0x065d7f (0x16f1 bytes) 000236[1] 1766388613.073454 read 0x048be6 thru 0x049b5b (0xf76 bytes) 000237[1] 1766388613.095043 mapread 0x1b4fdb thru 0x1b633a (0x1360 bytes) 000238[0] 1766388613.121416 mapwrite 0x5083ab thru 0x508eb5 (0xb0b bytes) 000239[0] 1766388613.179217 mapwrite 0x423705 thru 0x4245ab (0xea7 bytes) 000242[0] 1766388613.246282 write 0x70168a thru 0x70238c (0xd03 bytes) 000243[0] 1766388613.270203 write 0x8089f0 thru 0x80a85e (0x1e6f bytes) 000245[0] 1766388613.296827 mapread 0x7a4927 thru 0x7a4b2b (0x205 bytes) 000247[1] 1766388613.327680 mapwrite 0x82f7bd thru 0x830d54 (0x1598 bytes) 000248[0] 1766388613.426126 trunc from 0x9f018e to 0x536e8d (0x4b9300 bytes) 000249[1] 1766388613.470753 trunc from 0x536e8e to 0x5e6505 (0xaf678 bytes) 000250[1] 1766388613.504055 mapread 0x34d0cd thru 0x34ee94 (0x1dc8 bytes) 000251[0] 1766388613.511084 trunc from 0x5e6506 to 0x17dc45 (0x4688c0 bytes) 000254[1] 1766388613.608368 trunc from 0x17dc46 to 0x79dac0 (0x61fe7b bytes) 000262[0] 1766388613.684343 trunc from 0x79dac1 to 0x54426c (0x259854 bytes) 000263[0] 1766388613.713168 write 0x855bad thru 0x855cae (0x102 bytes) 000265[1] 1766388613.738069 mapread 0x3d1f0b thru 0x3d2d45 (0xe3b bytes) 000268[1] 1766388613.779635 mapwrite 0x1db2b1 thru 0x1dc1bf (0xf0f bytes) 000269[0] 1766388613.858868 mapwrite 0x691c5a thru 0x692e39 (0x11e0 bytes) 000271[0] 1766388613.993452 trunc from 0x855caf to 0x71db9a (0x138114 bytes) 000273[1] 1766388614.041253 mapwrite 0x418692 thru 0x41a59d (0x1f0c bytes) 000275[1] 1766388614.142569 write 0x49b3dd thru 0x49c24a (0xe6e bytes) 000278[0] 1766388614.214437 write 0x6dcd24 thru 0x6dd302 (0x5df bytes) 000280[0] 1766388614.264607 mapread 0x1480d2 thru 0x149adf (0x1a0e bytes) 000281[0] 1766388614.277722 mapwrite 0x39cd88 thru 0x39d0da (0x353 bytes) 000283[1] 1766388614.377429 read 0x700847 thru 0x700ad3 (0x28d bytes) 000284[0] 1766388614.395399 mapwrite 0x6492c3 thru 0x649609 (0x347 bytes) 000286[1] 1766388614.464131 read 0x304de2 thru 0x3051ec (0x40b bytes) 000287[1] 1766388614.480684 trunc from 0x71db9b to 0x1228c5 (0x5fb2d5 bytes) 000289[0] 1766388614.536024 read 0x06709d thru 0x067643 (0x5a7 bytes) 000290[0] 1766388614.569753 mapread 0x075657 thru 0x076b1e (0x14c8 bytes) 000291[0] 1766388614.609884 trunc from 0x1228c6 to 0x013c7d (0x10ec48 bytes) 000292[1] 1766388614.663475 read 0x00f4fe thru 0x01015c (0xc5f bytes) 000293[0] 1766388614.708609 read 0x012354 thru 0x013c69 (0x1916 bytes) 000295[1] 1766388614.735020 write 0x9af509 thru 0x9b0b1e (0x1616 bytes) 000296[1] 1766388614.769607 write 0x5f70b7 thru 0x5f75c6 (0x510 bytes) 000297[1] 1766388614.825712 write 0x1808ce thru 0x182848 (0x1f7b bytes) 000298[0] 1766388614.843950 write 0x0eb87c thru 0x0ebd2f (0x4b4 bytes) 000299[1] 1766388614.943033 write 0x4096c6 thru 0x40ae77 (0x17b2 bytes) 000300[1] 1766388614.987288 mapread 0x4cabf0 thru 0x4cbf0c (0x131d bytes) 000301[0] 1766388615.004980 mapread 0x6d6b0e thru 0x6d8389 (0x187c bytes) 000305[1] 1766388615.110571 mapread 0x3bb92a thru 0x3bba3a (0x111 bytes) 000307[0] 1766388615.160892 read 0x52b4f2 thru 0x52b806 (0x315 bytes) 000316[1] 1766388615.164181 trunc from 0x9b0b1f to 0x97d896 (0x33288 bytes) 000317[1] 1766388615.236601 mapwrite 0x13f2a3 thru 0x1406c5 (0x1423 bytes) 000319[1] 1766388615.298282 mapwrite 0x2572c2 thru 0x2580bd (0xdfc bytes) 000321[1] 1766388615.355175 read 0x2963c9 thru 0x296b24 (0x75c bytes) 000322[0] 1766388615.387569 read 0x39f29b thru 0x39fe6a (0xbd0 bytes) 000324[1] 1766388615.433425 mapread 0x00a1ef thru 0x00bceb (0x1afd bytes) 000326[1] 1766388615.490763 mapread 0x534d4c thru 0x534ff6 (0x2ab bytes) 000327[0] 1766388615.494748 write 0x0bfeeb thru 0x0c0f19 (0x102f bytes) 000328[1] 1766388615.532665 read 0x726867 thru 0x72884a (0x1fe4 bytes) 000335[1] 1766388615.700075 mapread 0x6fe28f thru 0x6ffe2f (0x1ba1 bytes) 000336[1] 1766388615.745121 mapread 0x8b7cc9 thru 0x8b8d81 (0x10b9 bytes) 000340[0] 1766388615.869940 read 0x21feb4 thru 0x220e4b (0xf98 bytes) 000342[0] 1766388615.873881 mapread 0x3199b2 thru 0x31b381 (0x19d0 bytes) 000346[1] 1766388615.957129 write 0x7ea42b thru 0x7eb721 (0x12f7 bytes) 000348[1] 1766388615.994453 trunc from 0x97d897 to 0x5f8281 (0x385615 bytes) 000350[1] 1766388616.031093 trunc from 0x5f8282 to 0x647f59 (0x4fcd8 bytes) 000351[1] 1766388616.045669 trunc from 0x647f5a to 0x38dfc3 (0x2b9f96 bytes) 000352[1] 1766388616.065586 write 0x03c95d thru 0x03d4ba (0xb5e bytes) 000353[1] 1766388616.082886 write 0x56e78c thru 0x56f492 (0xd07 bytes) 000354[1] 1766388616.086313 trunc from 0x56f493 to 0x03ec21 (0x530871 bytes) 000358[1] 1766388616.163973 write 0x630430 thru 0x631107 (0xcd8 bytes) 000360[0] 1766388616.182604 write 0x0389d2 thru 0x03a9b5 (0x1fe4 bytes) 000363[1] 1766388616.246461 write 0x17ea17 thru 0x17f484 (0xa6e bytes) 000365[0] 1766388616.293022 mapwrite 0x1ab5b5 thru 0x1ac1e9 (0xc35 bytes) 000366[1] 1766388616.401923 write 0x951670 thru 0x953417 (0x1da8 bytes) 000368[0] 1766388616.416167 trunc from 0x953418 to 0x6e7ac6 (0x26b951 bytes) 000373[0] 1766388616.475199 write 0x25feeb thru 0x2601c3 (0x2d9 bytes) 000374[0] 1766388616.487652 write 0x7df44d thru 0x7e04a2 (0x1056 bytes) 000375[1] 1766388616.505712 trunc from 0x7e04a3 to 0x93ef4e (0x15eaac bytes) 000376[0] 1766388616.563336 write 0x608cde thru 0x60a874 (0x1b97 bytes) 000380[1] 1766388616.639469 write 0x842773 thru 0x84411b (0x19a9 bytes) 000381[0] 1766388616.661129 read 0x5c8a0d thru 0x5ca170 (0x1764 bytes) 000384[0] 1766388616.693942 mapread 0x73bb40 thru 0x73d38e (0x184f bytes) 000385[0] 1766388616.719726 mapwrite 0x23da47 thru 0x23eeeb (0x14a5 bytes) 000386[1] 1766388616.782599 trunc from 0x93ef4f to 0x2e4f34 (0x65a01a bytes) 000387[0] 1766388616.851680 trunc from 0x2e4f35 to 0x678a7a (0x393b46 bytes) 000389[0] 1766388616.936458 mapread 0x2e3962 thru 0x2e5141 (0x17e0 bytes) 000392[1] 1766388616.992715 mapwrite 0x2bcf32 thru 0x2bdfd7 (0x10a6 bytes) 000393[0] 1766388617.076472 write 0x0f843e thru 0x0f985d (0x1420 bytes) 000395[1] 1766388617.110341 mapread 0x47812c thru 0x47a09d (0x1f72 bytes) 000396[0] 1766388617.138827 mapwrite 0x52ac10 thru 0x52aca8 (0x99 bytes) 000397[1] 1766388617.202454 write 0x2b4c40 thru 0x2b6387 (0x1748 bytes) 000398[0] 1766388617.242804 mapwrite 0x155f80 thru 0x157308 (0x1389 bytes) 000400[1] 1766388617.322422 write 0x615325 thru 0x617251 (0x1f2d bytes) 000401[0] 1766388617.350165 trunc from 0x678a7b to 0x56267c (0x1163fe bytes) 000402[0] 1766388617.389777 mapwrite 0x25ae1e thru 0x25af36 (0x119 bytes) 000405[1] 1766388617.429210 trunc from 0x56267d to 0x89f768 (0x33d0ec bytes) 000408[1] 1766388617.505070 mapread 0x596b00 thru 0x59880f (0x1d10 bytes) 000411[1] 1766388617.539133 read 0x606047 thru 0x607a78 (0x1a32 bytes) 000412[1] 1766388617.548994 mapwrite 0x4873fb thru 0x488924 (0x152a bytes) 000416[0] 1766388617.647600 read 0x0c9a02 thru 0x0c9ee0 (0x4df bytes) 000418[0] 1766388617.688419 mapwrite 0x43fab2 thru 0x440707 (0xc56 bytes) 000426[1] 1766388617.815122 trunc from 0x89f769 to 0x23e61d (0x66114b bytes) 000427[0] 1766388617.869292 mapwrite 0x0110ae thru 0x011431 (0x384 bytes) 000428[0] 1766388617.935109 read 0x1e3270 thru 0x1e48e6 (0x1677 bytes) 000429[0] 1766388618.000580 write 0x2043c5 thru 0x204b25 (0x761 bytes) 000433[0] 1766388618.056929 mapwrite 0x92a352 thru 0x92ab83 (0x832 bytes) 000434[1] 1766388618.172723 write 0x1a712f thru 0x1a8d4a (0x1c1c bytes) 000439[1] 1766388618.270795 write 0x74a367 thru 0x74ab8c (0x826 bytes) 000441[1] 1766388618.303174 mapread 0x5bc955 thru 0x5bde13 (0x14bf bytes) 000443[1] 1766388618.322291 mapread 0x4a9df7 thru 0x4abbe9 (0x1df3 bytes) 000444[1] 1766388618.335877 write 0x612b07 thru 0x613074 (0x56e bytes) 000448[0] 1766388618.358998 mapwrite 0x3d95f2 thru 0x3d96b8 (0xc7 bytes) 000449[1] 1766388618.410799 trunc from 0x92ab84 to 0x074806 (0x8b637d bytes) 000452[0] 1766388618.436900 read 0x00281e thru 0x003b3e (0x1321 bytes) 000453[1] 1766388618.466750 trunc from 0x074807 to 0x26f7d5 (0x1fafcf bytes) 000455[0] 1766388618.550000 mapread 0x1a9a61 thru 0x1a9db0 (0x350 bytes) 000456[1] 1766388618.597167 mapwrite 0x0db4fa thru 0x0dc695 (0x119c bytes) 000460[0] 1766388618.710582 trunc from 0x26f7d6 to 0x18cad3 (0xe2d02 bytes) 000461[1] 1766388618.746459 mapwrite 0x6a4e7c thru 0x6a58c6 (0xa4b bytes) 000462[0] 1766388618.828775 mapwrite 0x52f4d3 thru 0x52f7d6 (0x304 bytes) 000463[1] 1766388618.900614 read 0x508d5f thru 0x509f4f (0x11f1 bytes) 000466[0] 1766388618.958758 trunc from 0x6a58c7 to 0x14f5fd (0x5562c9 bytes) 000470[1] 1766388619.006863 mapwrite 0x923df2 thru 0x92563c (0x184b bytes) 000471[0] 1766388619.088815 read 0x6edf5a thru 0x6ee881 (0x928 bytes) 000473[1] 1766388619.124258 read 0x6184cb thru 0x619e5f (0x1995 bytes) 000475[1] 1766388619.145795 read 0x6f0b40 thru 0x6f2692 (0x1b53 bytes) 000478[1] 1766388619.187868 write 0x8caea6 thru 0x8ccb58 (0x1cb3 bytes) 000479[1] 1766388619.197527 mapwrite 0x40883c thru 0x40a546 (0x1d0b bytes) 000480[0] 1766388619.256369 mapread 0x2f9fd6 thru 0x2fb0e1 (0x110c bytes) 000481[0] 1766388619.357380 trunc from 0x92563d to 0x2e9411 (0x63c22b bytes) 000485[0] 1766388619.422971 mapwrite 0x30ba55 thru 0x30d1a3 (0x174f bytes) 000490[0] 1766388619.572709 read 0x30c0e4 thru 0x30ca27 (0x944 bytes) 000491[1] 1766388619.573584 mapwrite 0x2c66ba thru 0x2c6f3b (0x882 bytes) 000492[0] 1766388619.649059 write 0x91e645 thru 0x920543 (0x1eff bytes) 000494[1] 1766388619.676927 read 0x79f378 thru 0x79f8ba (0x543 bytes) 000496[0] 1766388619.696837 trunc from 0x920544 to 0x7a80ec (0x178457 bytes) 000497[1] 1766388619.722997 mapwrite 0x000244 thru 0x001cd8 (0x1a95 bytes) 000500[0] 1766388619.833810 mapread 0x2213d7 thru 0x22220c (0xe36 bytes) All operations completed A-OK! PASS 16b (18s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:30:22 (1766388622) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:30:24 (1766388624) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.176245 s, 47.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.178463 s, 47.0 MB/s ldlm.namespaces.MGC192.168.202.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff969d48441000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff969d59987800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff969d59987800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff969d59987800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.10386 s, 14.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.07892 s, 50.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.34218 s, 78.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.55026 s, 29.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3876 s, 19.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.97344 s, 53.1 MB/s PASS 16d (29s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:30:53 (1766388653) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110976 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647362 s, 16.2 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:30:56 (1766388656) TIMEOUT PASS 16f (23s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:31:20 (1766388680) TIMEOUT PASS 16g (24s) == sanityn test 16h: mmap read after truncate file ======= 02:31:43 (1766388703) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 02:31:46 (1766388706) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 02:31:49 (1766388709) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.916439 s, 28.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.476749 s, 55.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.919195 s, 28.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.785816 s, 33.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08333 s, 24.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.959925 s, 27.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.25411 s, 20.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.658395 s, 39.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.783932 s, 33.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.43189 s, 18.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52072 s, 17.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.881788 s, 29.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.976938 s, 26.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.12506 s, 23.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.16985 s, 22.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.16368 s, 22.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21815 s, 21.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.45188 s, 18.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51224 s, 17.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (16s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:32:05 (1766388725) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Mon Dec 22 02:32:23 EST 2025 + fsx 2 done: rc=0 Mon Dec 22 02:32:23 EST 2025 + fsx 3 done: rc=0 Mon Dec 22 02:32:23 EST 2025 + fsx 4 done: rc=0 Mon Dec 22 02:32:23 EST 2025 + fsx 5 done: rc=0 Mon Dec 22 02:32:23 EST 2025 PASS 16k (23s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:32:28 (1766388748) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 02:32:34 (1766388754) mmap test1: basic mmap operation (PASS, 0.057477s) mmap test2: MAP_PRIVATE not write back (PASS, 0.032766s) mmap test3: concurrent mmap ops on two nodes (PASS, 21.176s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1546s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.062401s) mmap test6: check mmap write/read content on two nodes (PASS, 0.076725s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.45521s) mmap test11: multi-thread shared mmap (PASS, 0.45758s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0287s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:33:05 (1766388785) oleg217-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:33:07 (1766388787) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:33:10 (1766388790) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 02:33:12 (1766388792) now is 1766388793 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 new atime is 1766388854 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:34:17 (1766388857) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:34:20 (1766388860) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:34:22 (1766388862) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (4s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:34:26 (1766388866) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 02:34:27 (1766388867) utime: good mknod times 1766388867 <= 1766388868 <= 1766388868 for /mnt/lustre/f26a utime: good mknod times 1766388867 <= 1766388868 <= 1766388868 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 02:34:31 (1766388871) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:34:36 (1766388876) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:34:37.000000000 -0500 Birth: 2025-12-22 02:34:37.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:34:37.000000000 -0500 Birth: 2025-12-22 02:34:37.000000000 -0500 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:34:41 (1766388881) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:34:45 (1766388885) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:34:49 (1766388889) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.117@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 02:35:02 (1766388902) -rw-r--r-- 1 root root 0 Dec 22 02:35 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:35:08 (1766388908) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42838 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:35:12 (1766388912) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:35:17 (1766388917) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:35:18 (1766388918) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:35:19 (1766388919) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:35:21 (1766388921) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 02:35:22 (1766388922) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid in IDLE state after 0 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid in FULL state after 0 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid in IDLE state after 0 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid in IDLE state after 0 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969d48441000.ost_server_uuid in IDLE state after 0 sec oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969d48441000.ost_server_uuid in IDLE state after 0 sec PASS 34 (97s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:36:59 (1766389019) fail_loc=0x80000317 Race attempt 0 Wait for 48958 49090 for 60 sec... - open/close 1595 (time 1766389032.60 total 10.01 last 159.42) - open/close 3322 (time 1766389042.60 total 20.01 last 172.68) total: 4000 open/close in 23.88 seconds: 167.48 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.117@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.117@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:38:05 (1766389085) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.493889 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.462636 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.49611 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.489375 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.472561 s, 111 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.483379 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.460871 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.456167 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.455528 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.482865 s, 109 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.968678 s, 54.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7443456) after(7494656) PASS 36 (242s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:42:08 (1766389328) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 - create 1172 (time 1766389342.43 total 10.00 last 117.15) - create 2908 (time 1766389352.43 total 20.00 last 173.60) - create 4703 (time 1766389362.43 total 30.00 last 179.49) - create 6652 (time 1766389372.43 total 40.01 last 194.89) - create 9152 (time 1766389382.43 total 50.01 last 249.92) total: 10000 create in 53.14 seconds: 188.18 ops/second PASS 37 (62s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:43:09 (1766389389) repeat after cancel_lru_locks PASS 39a (4s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:43:13 (1766389393) repeat after cancel_lru_locks PASS 39b (5s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:43:18 (1766389398) repeat after cancel_lru_locks PASS 39c (6s) == sanityn test 39d: sync write should update mtime ====== 02:43:24 (1766389404) fail_loc=0x411 fail_loc=0 PASS 39d (4s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:43:29 (1766389409) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (12s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:43:41 (1766389421) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:43:51 (1766389431) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (11s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:44:02 (1766389442) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (9s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:44:11 (1766389451) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (7s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:44:18 (1766389458) open: File exists Conflict PASS 41b (8s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:44:26 (1766389466) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (7s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:44:33 (1766389473) Conflict PASS 41d (7s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:44:40 (1766389480) Conflict PASS 41e (8s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:44:48 (1766389488) Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:44:56 (1766389496) Conflict PASS 41g (9s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:45:05 (1766389505) Conflict PASS 41h (9s) == sanityn test 41i: reint_open: create vs create ======== 02:45:14 (1766389514) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (937s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:00:51 (1766390451) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (8s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:01:00 (1766390460) open: File exists Conflict PASS 42b (9s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:01:09 (1766390469) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (9s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:01:18 (1766390478) Conflict PASS 42d (8s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:01:26 (1766390486) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (8s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:01:34 (1766390494) Conflict PASS 42f (9s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:01:43 (1766390503) Conflict PASS 42g (8s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:01:51 (1766390511) Conflict PASS 42h (9s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:02:00 (1766390520) PASS 43a (45s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 03:02:46 (1766390566) Conflict PASS 43b (9s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 03:02:55 (1766390575) Conflict PASS 43c (9s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 03:03:04 (1766390584) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (10s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 03:03:14 (1766390594) Conflict PASS 43e (9s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 03:03:23 (1766390603) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (7s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 03:03:30 (1766390610) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (8s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 03:03:38 (1766390618) Conflict PASS 43h (8s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:03:46 (1766390626) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:03:49 (1766390629) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (88s) == sanityn test 43k: unlink vs create ==================== 03:05:17 (1766390717) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (626s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:15:44 (1766391344) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Dec 22 03:15:48 EST 2025 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:15:50 (1766391350) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:15:56 (1766391356) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:16:03 (1766391363) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:16:10 (1766391370) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:16:17 (1766391377) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:16:24 (1766391384) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:16:31 (1766391391) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:16:37 (1766391397) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:16:39 (1766391399) PASS 45a (52s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:17:32 (1766391452) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:17:38 (1766391458) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:17:44 (1766391464) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:17:51 (1766391471) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:17:57 (1766391477) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:18:03 (1766391483) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:18:10 (1766391490) Conflict PASS 45h (6s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:18:16 (1766391496) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 03:18:18 (1766391498) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (832s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:32:10 (1766392330) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (8s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:32:18 (1766392338) open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:32:26 (1766392346) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:32:33 (1766392353) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:32:40 (1766392360) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:32:48 (1766392368) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:32:55 (1766392375) Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:33:02 (1766392382) Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:33:10 (1766392390) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:33:12 (1766392392) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:33:13 (1766392393) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:33:15 (1766392395) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:33:17 (1766392397) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:33:18 (1766392398) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:33:20 (1766392400) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:33:21 (1766392401) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:33:23 (1766392403) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0143491 s, 714 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:33:31 (1766392411) 0+1 records in 0+1 records out 158 bytes copied, 0.0130483 s, 12.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260182) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:33:36 (1766392416) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00623442 s, 164 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371259 s, 28.2 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:33:52 (1766392432) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0170577 s, 60.0 kB/s PASS 51c (9s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:34:01 (1766392441) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207812 s, 50.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:34:05 (1766392445) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 03:34:11 (1766392451) llite.lustre-ffff969d48441000.xattr_cache=1 llite.lustre-ffff969d59987800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:11.000000000 -0500 Modify: 2025-12-22 03:34:11.000000000 -0500 Change: 2025-12-22 03:34:11.000000000 -0500 Birth: 2025-12-22 03:34:12.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:17.000000000 -0500 Modify: 2025-12-22 03:34:18.000000000 -0500 Change: 2025-12-22 03:34:18.000000000 -0500 Birth: 2025-12-22 03:34:18.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:23.000000000 -0500 Modify: 2025-12-22 03:34:23.000000000 -0500 Change: 2025-12-22 03:34:23.000000000 -0500 Birth: 2025-12-22 03:34:23.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:23.000000000 -0500 Modify: 2025-12-22 03:34:23.000000000 -0500 Change: 2025-12-22 03:34:23.000000000 -0500 Birth: 2025-12-22 03:34:23.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:28.000000000 -0500 Modify: 2025-12-22 03:34:29.000000000 -0500 Change: 2025-12-22 03:34:29.000000000 -0500 Birth: 2025-12-22 03:34:29.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:34:28.000000000 -0500 Modify: 2025-12-22 03:34:29.000000000 -0500 Change: 2025-12-22 03:34:29.000000000 -0500 Birth: 2025-12-22 03:34:29.000000000 -0500 PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 03:34:37 (1766392477) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 03:34:45 (1766392485) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 03:34:55 (1766392495) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 03:35:08 (1766392508) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:35:19 (1766392519) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:35:20 (1766392520) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 03:35:34 (1766392534) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (16s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:35:50 (1766392550) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1766392551 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.2457 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (53s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:36:43 (1766392603) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1766392605 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.97742 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (46s) == sanityn test 60: Verify data_version behaviour ======== 03:37:29 (1766392649) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 03:37:32 (1766392652) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:37:35 (1766392655) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:37:39 (1766392659) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00728549 s, 9.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:37:41 (1766392661) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0078973 s, 8.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:37:43 (1766392663) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:37:44 (1766392664) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00953544 s, 6.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:37:46 (1766392666) llite.lustre-ffff969d48441000.xattr_cache=1 llite.lustre-ffff969d59987800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:37:49 (1766392669) llite.lustre-ffff969d48441000.xattr_cache=1 llite.lustre-ffff969d59987800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:37:52 (1766392672) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00720581 s, 142 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00717852 s, 143 kB/s 272960: get lock2 272961: get lock1 272960: done 272960: get lock1 272961: done 272961: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 272961: cannot get lock2: Resource deadlock avoided 272961: exit rc=0 272960: done 272960: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:37:58 (1766392678) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0229848 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192451 s, 54.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273543 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 03:38:06 (1766392686) opening files: ........ closing files: ........ PASS 76 (27s) == sanityn test 77a: check FIFO NRS policy =============== 03:38:33 (1766392713) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169955 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608876 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649008 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792807 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659328 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596277 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798138 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904764 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957098 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830706 s, 12.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11177 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101356 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122142 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12753 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126692 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143915 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160409 s, 6.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182711 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285961 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176385 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198657 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223433 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203893 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262534 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230465 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260172 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026879 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292817 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031844 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297868 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303963 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320941 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363362 s, 28.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 03:38:37 (1766392717) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183636 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497895 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813504 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890999 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091262 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838843 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739492 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111875 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112524 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973107 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136677 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159654 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224311 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157908 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023407 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115766 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130938 s, 80.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180512 s, 5.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013482 s, 77.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269075 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20634 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196036 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233015 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116905 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212107 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00693074 s, 151 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321029 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018956 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007054 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158558 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205368 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259294 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375453 s, 279 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.182927 s, 91.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274237 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339914 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815901 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897023 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115496 s, 9.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108402 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124145 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109512 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128629 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136161 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139898 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12016 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133061 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125314 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150645 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143921 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172122 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345245 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245529 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380983 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124852 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260958 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313959 s, 33.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315674 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297459 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225341 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373672 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036056 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037187 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493725 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251539 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351416 s, 29.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=3 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 03:38:42 (1766392722) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193955 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325549 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348757 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108592 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105829 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108443 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120376 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123638 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119243 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110817 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144238 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13966 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151359 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149661 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135457 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145426 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144146 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145692 s, 72.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128694 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177897 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463481 s, 226 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473678 s, 221 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630696 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316766 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376261 s, 279 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988673 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572396 s, 183 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401156 s, 261 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307001 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293315 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264413 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043736 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201074 s, 52.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174337 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680115 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889195 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958995 s, 10.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892295 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913556 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794493 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896855 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957802 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777185 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982957 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11595 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108888 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995675 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106118 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11125 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116576 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243157 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206204 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155184 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298097 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246867 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212324 s, 49.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313879 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248667 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279514 s, 37.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229446 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295132 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025673 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026985 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237627 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310381 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302731 s, 34.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166956 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027076 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452148 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617919 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664654 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828274 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117712 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122524 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112914 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118949 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108348 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106165 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122173 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11668 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109814 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114444 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131585 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185926 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196967 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128051 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314349 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245585 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272503 s, 38.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240997 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275447 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208629 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213848 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249611 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252174 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276495 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237056 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288874 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025994 s, 40.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 03:38:49 (1766392729) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.221521 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465878 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050871 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631881 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674543 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599984 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054907 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430623 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387834 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253197 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375062 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354336 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375737 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460817 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650577 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514273 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579638 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607524 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635123 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669892 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660167 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065943 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675942 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674239 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082052 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21135 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20906 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211037 s, 5.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196558 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224936 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22545 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230179 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226114 s, 4.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190698 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359022 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419046 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108616 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964217 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115832 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117587 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129689 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12285 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144595 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140099 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142393 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139219 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166886 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148344 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149629 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169937 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224166 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199674 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066941 s, 157 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287177 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561019 s, 187 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304909 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306423 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475818 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319431 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315489 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332515 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034447 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356288 s, 29.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386038 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354275 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607963 s, 17.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169485 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036137 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600559 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721788 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929001 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853758 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924745 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818353 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883853 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110356 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918571 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981037 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106343 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112089 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102789 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117607 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112481 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228223 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187211 s, 56.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157122 s, 66.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00762314 s, 138 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266228 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111083 s, 94.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238926 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233571 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237994 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422904 s, 24.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03021 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394357 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376141 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180321 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219214 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126892 s, 82.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 03:38:55 (1766392735) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.17@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166496 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265348 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449831 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844138 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989736 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092904 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902593 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879124 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881289 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111001 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979984 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111474 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11892 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121606 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966799 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128579 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11602 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197205 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20234 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199931 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205174 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201781 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208493 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199611 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20661 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200992 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200392 s, 5.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203456 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215837 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209884 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208403 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207791 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209826 s, 5.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1676 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038153 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268831 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110889 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121894 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128958 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133122 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119838 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133645 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122011 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138167 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124501 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134621 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131606 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133139 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116468 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144931 s, 7.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190617 s, 5.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201237 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192311 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195573 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195822 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196489 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207323 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198249 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204008 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206107 s, 5.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204897 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2116 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209062 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203408 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209679 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213561 s, 4.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.194178 s, 86.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533702 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536123 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619022 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610898 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769806 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852549 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737458 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514745 s, 20.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979743 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744388 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725877 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761883 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441788 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102066 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096688 s, 108 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257898 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205283 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159012 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222133 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221702 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142688 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319551 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249186 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252905 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749172 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142964 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479823 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00558449 s, 188 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529009 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049475 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526154 s, 19.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.178964 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617395 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808795 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699328 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969373 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895261 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908095 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110865 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924836 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136639 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149967 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114336 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151626 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176043 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157914 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182525 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173456 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00317419 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00313987 s, 334 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262057 s, 400 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293389 s, 357 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875537 s, 12.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925376 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808538 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346846 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256973 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295219 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015907 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146913 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103416 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813642 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602142 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927621 s, 113 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:39:05 (1766392745) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174065 s, 96.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388875 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368067 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583854 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527095 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631282 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558731 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664244 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461218 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483172 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895712 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972483 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135901 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133776 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122577 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145069 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143081 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808806 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079581 s, 132 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00854444 s, 123 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089049 s, 118 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281447 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347444 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331351 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434736 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270818 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048382 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468176 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545445 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119049 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111434 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124926 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474332 s, 221 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169797 s, 98.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026685 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335644 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908024 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994353 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108731 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124619 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103908 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102308 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12628 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104725 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132909 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125386 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122532 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121164 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134656 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120766 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00939377 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183344 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119721 s, 87.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129753 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012272 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135254 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164671 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305898 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017288 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022439 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209093 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020832 s, 50.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218811 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242919 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253491 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246088 s, 42.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201528 s, 83.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488138 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477761 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108093 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126668 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125886 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111214 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126667 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123818 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108419 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124362 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134468 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141044 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126823 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114708 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124577 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129693 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870092 s, 121 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134792 s, 77.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134775 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117354 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277273 s, 37.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133374 s, 78.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152197 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242234 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168013 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195841 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203811 s, 51.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279401 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301267 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223902 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540765 s, 194 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273851 s, 38.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158226 s, 106 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423567 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374741 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610947 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071478 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538057 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06028 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843593 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10873 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703123 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153289 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129361 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149412 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157498 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138521 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111551 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15622 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017453 s, 60.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136687 s, 76.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146959 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200752 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217824 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292915 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224323 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316339 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280906 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241572 s, 43.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225351 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296181 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320595 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398694 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445545 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346019 s, 30.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 03:39:15 (1766392755) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:39:20 (1766392760) oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg217-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg217-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg217-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg217-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 03:39:25 (1766392765) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:39:35 (1766392775) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163561 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347379 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349471 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921692 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103742 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111541 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112615 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109432 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10073 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10572 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111383 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116665 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126176 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129426 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114744 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126751 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12818 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158045 s, 66.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253854 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298728 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179864 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204036 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249152 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305858 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254082 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258612 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267885 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03526 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300932 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304282 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306577 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320754 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346005 s, 30.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23182 s, 20.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4795 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (50s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:40:24 (1766392824) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25528 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645998 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742909 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105289 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748795 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108327 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1154 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830541 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093799 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133423 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107588 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138382 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160195 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160276 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137323 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184664 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153955 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235134 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263206 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238306 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249169 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025468 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266729 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199009 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311102 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266436 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254287 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349801 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297275 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335145 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243626 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283198 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361932 s, 29.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7358 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.902 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23948 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403087 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667674 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661567 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702821 s, 14.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815727 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866376 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104521 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931948 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136766 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136898 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159483 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174602 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179529 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197975 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206542 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214675 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196863 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307793 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240305 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332778 s, 31.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312414 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291351 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332077 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343601 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360766 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322727 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327308 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367311 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301627 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413086 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450228 s, 23.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8601 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9694 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (139s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:42:43 (1766392963) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24168 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369482 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039199 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504989 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730988 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691449 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824586 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744226 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587834 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153766 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198174 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1968 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233389 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239957 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212893 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243179 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24452 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248853 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253356 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293335 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295137 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317307 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221446 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033257 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261357 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352901 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316899 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290334 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289677 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382198 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332616 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331903 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362049 s, 29.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7382 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9445 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23818 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506154 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447661 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071457 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771257 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814676 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589775 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653478 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787541 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140587 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149273 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169853 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157977 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173696 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205026 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211419 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230022 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030017 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033927 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304697 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364887 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307871 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402814 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387922 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029634 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362515 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357835 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420727 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344491 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045095 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437975 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395316 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418444 s, 25.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7497 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.971 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (137s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:45:00 (1766393100) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275352 s, 60.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384132 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351716 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567042 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601768 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05607 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812048 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781533 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888904 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809851 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712872 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113325 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071732 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904766 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119105 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115521 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987539 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209577 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170688 s, 61.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187551 s, 55.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172001 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188821 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187424 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215943 s, 48.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247447 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218768 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209761 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346391 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239716 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257136 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025502 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026494 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275111 s, 38.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0378 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77501 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.17@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.17@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.182405 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306389 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277479 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880012 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106417 s, 9.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764608 s, 13.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823874 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857873 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11641 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112485 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116578 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119324 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11477 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128868 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121089 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125116 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118302 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353166 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247437 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338044 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296847 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343471 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352736 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325923 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354673 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353864 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264164 s, 39.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325693 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369998 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400597 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400246 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372629 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375751 s, 27.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0324 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77349 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.17@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181355 s, 92.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218772 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400074 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385505 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104618 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968087 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113391 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125312 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124305 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124096 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110194 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122428 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133535 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129757 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114692 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128413 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148613 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271115 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178065 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01086 s, 96.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00841551 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405596 s, 259 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181674 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139604 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201383 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265532 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015906 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335088 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394635 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220211 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563177 s, 186 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541708 s, 194 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540994 s, 194 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09109 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90513 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.483622 s, 34.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236254 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452076 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743144 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848342 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875061 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975201 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987084 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834237 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114186 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139918 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12152 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127872 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159281 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133967 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188139 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213023 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185485 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283038 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146022 s, 71.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153112 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187138 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347473 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203776 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295933 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209238 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023035 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265844 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236987 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246598 s, 42.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249634 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297572 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296938 s, 35.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96541 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7637 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99568 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76172 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154532 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307307 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356009 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846386 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733289 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689004 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796997 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109081 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105097 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114558 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121093 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113206 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120361 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124383 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116559 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125763 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118352 s, 8.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021487 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215383 s, 48.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213114 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218845 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203621 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227262 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240254 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248256 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252934 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413027 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324126 s, 32.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031731 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273859 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303501 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462625 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346642 s, 30.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.535 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4854 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.189359 s, 88.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300871 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493535 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598233 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644652 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497786 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705074 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409728 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943502 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10061 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114568 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962303 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119487 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120433 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105294 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137437 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13584 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182731 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187843 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207162 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204304 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223776 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225206 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229419 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029682 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264751 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025931 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028145 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463726 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449554 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306865 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304665 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573096 s, 18.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99746 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78409 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.170322 s, 98.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355749 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048154 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758626 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604897 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967507 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896878 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913706 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08189 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790697 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103894 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761126 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941754 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100646 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901918 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106404 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930514 s, 11.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134918 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903579 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00984415 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225694 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012224 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278895 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194125 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222111 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169834 s, 61.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175989 s, 59.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207726 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205982 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220751 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235417 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237341 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239869 s, 43.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85313 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81092 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80815 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75181 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (357s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:50:57 (1766393457) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.293706 s, 57.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435609 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408118 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150019 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124956 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146467 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179311 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167259 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148615 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162764 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165355 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172987 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169168 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167674 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16119 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186893 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175062 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216797 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135999 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217807 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210317 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224667 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204504 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241497 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025163 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220267 s, 47.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225821 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291257 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02232 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250019 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261722 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257374 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065872 s, 159 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15006 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.763 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (39s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:51:36 (1766393496) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.17@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.17@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143132 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395858 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292682 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553671 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583257 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288187 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643776 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460115 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090716 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815088 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909026 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891318 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105516 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934271 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682307 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108604 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749518 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221716 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163789 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202943 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174431 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148837 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217225 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265445 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279675 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290282 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180902 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024934 s, 42.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192472 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298124 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265036 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283649 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280671 s, 37.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11005 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7498 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:52:17 (1766393537) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.17@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.283507 s, 59.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354811 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309288 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107945 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10257 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158475 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145571 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164165 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169976 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170708 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161094 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160399 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16153 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154117 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156146 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155205 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161864 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115856 s, 90.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144305 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280535 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261534 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232628 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01662 s, 63.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172389 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017601 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184152 s, 56.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186255 s, 56.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203684 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228712 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229089 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234623 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242542 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249181 s, 42.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12098 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99236 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:52:49 (1766393569) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16389 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491679 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545645 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554011 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446108 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545991 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460386 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670326 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525988 s, 19.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526896 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873797 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970968 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102761 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101199 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112346 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112353 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919746 s, 11.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196204 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150163 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130148 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205054 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325328 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226708 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250144 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193633 s, 54.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265993 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023943 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171933 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645462 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436947 s, 240 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332257 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282927 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868103 s, 121 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86026 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76352 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9397 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75172 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (72s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:54:01 (1766393641) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135363 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280592 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565529 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467377 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724944 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541069 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686143 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071945 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774857 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655938 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810873 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106853 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104293 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11838 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097118 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129634 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134356 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170177 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159823 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178902 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222697 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248946 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145197 s, 72.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167361 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612524 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222681 s, 47.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870323 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262627 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260305 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241389 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145794 s, 71.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288142 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236584 s, 44.3 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5431 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9498 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (64s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:55:05 (1766393705) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14876 s, 113 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055804 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690347 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816541 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874763 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100009 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11031 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099613 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128597 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101198 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130496 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997467 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111198 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106629 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118004 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103668 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114313 s, 9.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130639 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202014 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134081 s, 78.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127714 s, 82.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167514 s, 62.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175698 s, 59.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190639 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249185 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026476 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208177 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0054497 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057438 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110405 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888062 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449947 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578858 s, 181 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.07423 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74595 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg217-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133591 s, 126 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515513 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713363 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748866 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115463 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939815 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110814 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109674 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125792 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11727 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134164 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126189 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122357 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136716 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120516 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137988 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154167 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128692 s, 81.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166222 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423378 s, 248 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180424 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022273 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022409 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189098 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024022 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374733 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210927 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308374 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294784 s, 35.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223822 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265953 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044626 s, 23.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg217-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg217-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79232 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74516 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77002 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74586 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (118s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:57:03 (1766393823) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:57:06 (1766393826) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg217-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.55757 s, 295 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg217-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0185 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:57:57 (1766393877) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.15257 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352977 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331957 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104485 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128849 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135894 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122689 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12204 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147432 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff969d48441000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff969d48441000.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124698 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150186 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142988 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301147 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412307 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262589 s, 39.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199215 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357598 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929928 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417827 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306089 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039934 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357206 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509935 s, 20.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156052 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351139 s, 299 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162328 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189161 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169548 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199748 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234167 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172387 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273805 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169259 s, 6.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97278 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97718 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139758 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03884 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208181 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603716 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897832 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985117 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941638 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848984 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093337 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12146 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116168 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111027 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104109 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108591 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117217 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130063 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120236 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111434 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020032 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163919 s, 64.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159217 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209937 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195779 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239049 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170901 s, 61.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265291 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577908 s, 181 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445802 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284906 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474091 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363417 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00342941 s, 306 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg217-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg217-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94489 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97419 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (70s) == sanityn test 77o: Changing rank should not panic ====== 03:59:07 (1766393947) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (9s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:59:16 (1766393956) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (58s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:00:14 (1766394014) mds.MDS.mdt.nrs_policies=tbf oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg217-client: oleg217-server: ssh exited with exit code 36 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg217-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (15s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:00:29 (1766394029) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 308852 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:01:09 (1766394069) oleg217-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg217-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg217-client: oleg217-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 04:01:13 (1766394073) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506302 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:01:14.000000000 -0500 Modify: 2025-12-22 04:01:14.000000000 -0500 Change: 2025-12-22 04:01:14.000000000 -0500 Birth: 2025-12-22 04:01:15.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (14s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:01:27 (1766394087) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:01:28 (1766394088) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:01:29 (1766394089) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:01:31 (1766394091) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:01:32 (1766394092) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:01:34 (1766394094) - open/close 1658 (time 1766394106.41 total 10.00 last 165.78) total: 2500 open/close in 14.04 seconds: 178.04 ops/second snapshot_time 1766394133.451814518 secs.nsecs start_time 1766394095.935851928 secs.nsecs elapsed_time 37.515962590 secs.nsecs open 2505 samples [usecs] 133 6047 1927346 1699501626 close 2504 samples [usecs] 113 2368 685909 218477663 mknod 2501 samples [usecs] 347 5906 1701546 1360511970 unlink 2500 samples [usecs] 287 25431 2122854 7984736360 mkdir 2 samples [usecs] 1004 1191 2195 2426497 rmdir 2 samples [usecs] 2220 3846 6066 19720116 rename 2500 samples [usecs] 335 15044 4855646 14441629684 getattr 3760 samples [usecs] 7 219 89395 2947523 parallel_rename_file 2500 samples [usecs] 335 15044 4855646 14441629684 crossdir_rename 2500 samples [usecs] 335 15044 4855646 14441629684 PASS 81d (40s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:02:14 (1766394134) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7549 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7549 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:02:17 (1766394137) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:02:18 (1766394138) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 04:02:25 (1766394145) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1766394146 Finishing test test1 at 1766394146 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:02:28 (1766394148) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 04:02:29 (1766394149) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:02:30 (1766394150) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:02:31 (1766394151) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5618 0x15f2 0x280000400 0 5641 0x1609 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5642 0x160a 0x240000400 1 5619 0x15f3 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 04:02:40 (1766394160) osc.lustre-OST0000-osc-ffff969d48441000.idle_timeout=0 osc.lustre-OST0000-osc-ffff969d59987800.idle_timeout=0 osc.lustre-OST0001-osc-ffff969d48441000.idle_timeout=0 osc.lustre-OST0001-osc-ffff969d59987800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0360399 s, 227 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09499 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 326247 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011445 s, 358 kB/s osc.lustre-OST0000-osc-ffff969d48441000.idle_timeout=debug osc.lustre-OST0000-osc-ffff969d59987800.idle_timeout=debug osc.lustre-OST0001-osc-ffff969d48441000.idle_timeout=debug osc.lustre-OST0001-osc-ffff969d59987800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:02:50 (1766394170) llite.lustre-ffff969d48441000.fast_read=0 llite.lustre-ffff969d59987800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.019678 s, 107 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff969d48441000.fast_read=1 llite.lustre-ffff969d59987800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:03:05 (1766394185) llite.lustre-ffff969d48441000.fast_read=0 llite.lustre-ffff969d59987800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00179077 s, 6.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff969d48441000.fast_read=1 llite.lustre-ffff969d59987800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:03:13 (1766394193) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:03:14 (1766394194) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00218917 s, 1.9 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:03:16 (1766394196) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0472758 s, 44.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:03:18 (1766394198) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0496438 s, 42.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 04:03:21 (1766394201) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00173894 s, 6.9 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 04:03:23 (1766394203) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00061974 s, 6.6 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 04:03:25 (1766394205) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000482938 s, 8.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 04:03:28 (1766394208) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000630313 s, 6.5 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 04:03:32 (1766394212) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1766394213 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1766394215 Finishing test test23 at 1766394215 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:03:39 (1766394219) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 04:03:40 (1766394220) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 04:03:58 (1766394238) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:03:59 (1766394239) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.003217 s, 318 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508837 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:03:59.000000000 -0500 Modify: 2025-12-22 04:03:59.000000000 -0500 Change: 2025-12-22 04:03:59.000000000 -0500 Birth: 2025-12-22 04:04:00.000000000 -0500 /mnt/lustre/f106b.sanityn 144115205272508837 -rw-r--r-- 1 0 0 1766394240 1766394239 1766394239 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508837 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:03:59.000000000 -0500 Modify: 2025-12-22 04:03:59.000000000 -0500 Change: 2025-12-22 04:03:59.000000000 -0500 Birth: 2025-12-22 04:04:00.000000000 -0500 1766394239 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 04:04:01 (1766394241) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 04:04:03 (1766394243) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0978602 s, 107 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5625 0x15f9 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:04:08 (1766394248) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.102083 s, 103 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5651 0x1613 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7549 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 04:04:16 (1766394256) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00235588 s, 4.2 MB/s llite.lustre-ffff969d48441000.hybrid_io=0 llite.lustre-ffff969d59987800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.27758 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:04:23 (1766394263) 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre 192.168.202.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 PASS 109 (510s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:12:53 (1766394773) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:12:54.000000000 -0500 Modify: 2025-12-22 04:12:54.000000000 -0500 Change: 2025-12-22 04:12:54.000000000 -0500 Birth: 2025-12-22 04:12:54.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg217-client.virtnet oleg217-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:13:59 (1766394839) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 04:14:00 (1766394840) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 04:14:01 (1766394841) lustre-OST0000-osc-ffff969d46fa8000 active. lustre-OST0001-osc-ffff969d46fa8000 active. lustre-OST0000-osc-ffff969d46fa8000 active. lustre-OST0001-osc-ffff969d46fa8000 active. MGC192.168.202.117@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 04:14:03 (1766394843) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:14:04 (1766394844) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:14:06 (1766394846) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 04:14:07 (1766394847) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (3s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 04:14:10 (1766394850) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 04:14:11 (1766394851) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (1s) == sanityn test 117d: Per-User Trash can Type testing ==== 04:14:12 (1766394852) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 04:14:14 (1766394854) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (2s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 04:14:15 (1766394855) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 04:14:16 (1766394856) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 04:14:17 (1766394857) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (5s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 04:14:22 (1766394862) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (3s) == sanityn test 117j: Move back file from Trash Can ====== 04:14:25 (1766394865) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 04:14:26 (1766394866) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 04:14:27 (1766394867) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (1s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 04:14:29 (1766394869) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (1s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 04:14:30 (1766394870) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (1s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 04:14:31 (1766394871) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (1s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 04:14:32 (1766394872) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 04:14:33 (1766394873) SKIP: sanityn test_117pb needs >= 2 MDTs SKIP 117pb (1s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 04:14:34 (1766394874) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (1s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 04:14:35 (1766394875) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (1s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 04:14:36 (1766394876) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (1s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 04:14:38 (1766394878) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (1s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 04:14:39 (1766394879) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (1s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 04:14:40 (1766394880) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 04:14:41 (1766394881) SKIP: sanityn test_117v ldiskfs only test SKIP 117v (2s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 04:14:43 (1766394883) SKIP: sanityn test_117w ldiskfs only test SKIP 117w (1s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 04:14:44 (1766394884) SKIP: sanityn test_117x ldiskfs only test SKIP 117x (1s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 04:14:45 (1766394885) SKIP: sanityn test_117ya ldiskfs only test SKIP 117ya (1s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 04:14:46 (1766394886) SKIP: sanityn test_117yb ldiskfs only test SKIP 117yb (1s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 04:14:47 (1766394887) SKIP: sanityn test_117yc ldiskfs only test SKIP 117yc (1s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 04:14:48 (1766394888) SKIP: sanityn test_117za ldiskfs only test SKIP 117za (1s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 04:14:49 (1766394889) SKIP: sanityn test_117zb ldiskfs only test SKIP 117zb (2s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 04:14:51 (1766394891) SKIP: sanityn test_117zc ldiskfs only test SKIP 117zc (1s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 04:14:52 (1766394892) On MGS 192.168.202.117, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.202.117, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] SKIP: sanityn test_117zd ldiskfs only test On MGS 192.168.202.117, active = nodemap.active=0 waiting 10 secs for sync SKIP 117zd (37s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 04:15:29 (1766394929) SKIP: sanityn test_118Aa ldiskfs only test SKIP 118Aa (1s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 04:15:30 (1766394930) SKIP: sanityn test_118Ab ldiskfs only test SKIP 118Ab (1s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 04:15:31 (1766394931) SKIP: sanityn test_118Ba ldiskfs only test SKIP 118Ba (1s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 04:15:32 (1766394932) SKIP: sanityn test_118Bb ldiskfs only test SKIP 118Bb (1s) == sanityn test 118C: TCU: add support for rename() ====== 04:15:33 (1766394933) SKIP: sanityn test_118C ldiskfs only test SKIP 118C (2s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 04:15:35 (1766394935) SKIP: sanityn test_118d ldiskfs only test SKIP 118d (1s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 04:15:36 (1766394936) SKIP: sanityn test_118e ldiskfs only test SKIP 118e (2s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:15:38 (1766394938) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=19 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0841 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0959 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0897 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0993 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0944 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0922 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0973 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1004 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0979 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1523 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1687 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1281 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1705 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1517 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1602 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1779 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1353 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1445 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2115 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2679 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2382 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2168 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2163 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.216 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2718 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2287 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2543 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3375 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3459 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3097 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3076 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3202 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3541 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3114 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3406 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3423 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.421 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.386 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff969d4634a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969d4634a000.ost_server_uuid in IDLE state after 0 sec PASS 200 (106s) cleanup: ====================================================== == sanityn test complete, duration 6770 sec ============== 04:17:25 (1766395045) === sanityn: start cleanup 04:17:25 (1766395045) === Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg217-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:18:15 (1766395095) ===