-----============= acceptance-small: sanityn ============----- Sun Dec 7 05:24:59 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg318-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 05:25:19 (1765103119) === Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Mount client oleg318-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg318-client.virtnet: executing check_config_client /mnt/lustre oleg318-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg318-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c1bc6e15800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1bc6e95800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6e15800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6e95800.idle_timeout=debug disable quota as required oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 05:25:50 (1765103150) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173645 s, 60.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8405] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 05:25:52 (1765103152) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 05:25:59 (1765103159) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 05:26:07 (1765103167) -rw-r--r-- 1 root root 0 Dec 7 05:26 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 05:26:13 (1765103173) -rw-r--r-- 1 root root 0 Dec 7 05:26 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 05:26:19 (1765103179) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 05:26:26 (1765103186) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 05:26:32 (1765103192) total 1 -rwxrwxrwx 1 root root 0 Dec 7 05:26 f2a -rwxrwxrwx 1 root root 0 Dec 7 05:26 f2b total 1 -rwxrwxrwx 1 root root 0 Dec 7 05:26 f2a -rwxrwxrwx 1 root root 0 Dec 7 05:26 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0170038 s, 30.1 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 05:26:38 (1765103198) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (7s) == sanityn test 2g: check blocks update on sync write ==== 05:26:45 (1765103205) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.341223 s, 6.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 05:26:51 (1765103211) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 05:26:56 (1765103216) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 05:27:03 (1765103223) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 05:27:09 (1765103229) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 05:27:15 (1765103235) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 05:27:19 (1765103239) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 05:27:24 (1765103244) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 05:27:30 (1765103250) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762416 s, 13.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0403207 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 05:27:36 (1765103256) 1+0 records in 1+0 records out 1 byte copied, 0.0142601 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.015244 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0181233 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00975451 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0318315 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0148516 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0168662 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0159403 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0298301 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0243511 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0234707 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0152414 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 05:27:44 (1765103264) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0146692 s, 209 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0444225 s, 92.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000206893 s, 14.8 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 05:27:50 (1765103270) striped dir -i1 -c2 -H crush /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 05:27:57 (1765103277) warning: '-runtime' deprecated, use '-t runtime' instead - link 365 (time 1765103289.36 total 10.02 last 36.43) - link 754 (time 1765103299.39 total 20.05 last 38.78) - link 1138 (time 1765103309.40 total 30.06 last 38.37) - link 1511 (time 1765103319.41 total 40.07 last 37.26) - link 1861 (time 1765103329.42 total 50.08 last 34.96) - link 2218 (time 1765103339.43 total 60.10 last 35.64) - link 2607 (time 1765103349.45 total 70.11 last 38.83) - link 2996 (time 1765103359.45 total 80.11 last 38.89) - link 3378 (time 1765103369.47 total 90.14 last 38.12) total: 3742 link in 100.01 seconds: 37.41 ops/second using seed 1295530824 running for 100 seconds - stat 9681 (time 1765103291 ; total 11 ; last 11) - stat 10000 (time 1765103291 ; total 11 ; last 0) - stat 20000 (time 1765103300 ; total 20 ; last 9) - stat 30000 (time 1765103309 ; total 29 ; last 9) - stat 40000 (time 1765103319 ; total 39 ; last 10) - stat 50000 (time 1765103329 ; total 49 ; last 10) - stat 54702 (time 1765103340 ; total 60 ; last 11) - stat 60000 (time 1765103350 ; total 70 ; last 10) - stat 65250 (time 1765103361 ; total 81 ; last 11) - stat 70000 (time 1765103370 ; total 90 ; last 9) total: 74670 stats in 100 seconds: 746.700012 stats/second - unlinked 0 (time 1765103330 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3742) error: No such file or directory total: 3741 unlinks in 87 seconds: 43.000000 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22767) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22769) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 05:30:22 (1765103422) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 05:30:32 (1765103432) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 23977 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 05:30:40 (1765103440) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 24569 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 05:30:45 (1765103445) striped dir -i0 -c2 -H crush /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25159 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 05:30:52 (1765103452) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25753 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 05:30:59 (1765103459) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26347 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 05:31:07 (1765103467) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (44s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 05:31:51 (1765103511) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.783181 s, 10.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.612353 s, 13.7 MB/s Chance of close/open is 1 in 50 Seed set to 3515 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[0] 1765103516.064596 mapwrite 0x5fff8c thru 0x6012f6 (0x136b bytes) 000003[0] 1765103516.197758 mapread 0x5b6979 thru 0x5c627d (0xf905 bytes) 000008[0] 1765103516.342231 read 0x11a1fd thru 0x122801 (0x8605 bytes) 000009[0] 1765103516.355422 mapread 0x12af94 thru 0x136056 (0xb0c3 bytes) 000010[1] 1765103516.380424 mapwrite 0x077332 thru 0x0837d1 (0xc4a0 bytes) 000012[1] 1765103516.534396 mapwrite 0x8d274e thru 0x8e2191 (0xfa44 bytes) 000014[1] 1765103516.694561 read 0x3b95b6 thru 0x3c2029 (0x8a74 bytes) 000017[0] 1765103516.723616 write 0x8eac8b thru 0x8f7d09 (0xd07f bytes) 000018[0] 1765103516.788663 mapwrite 0x37a731 thru 0x3823d0 (0x7ca0 bytes) truncating to largest ever: 0x9e782e 000019[0] 1765103516.992950 trunc from 0x8f7d0a to 0x9e782d (0xefb24 bytes) 000022[0] 1765103517.093836 trunc from 0x9e782e to 0x49dc4e (0x549bdf bytes) 000025[0] 1765103517.159047 write 0x58c804 thru 0x59600e (0x980b bytes) 000027[0] 1765103517.165797 write 0x6aa06c thru 0x6ac1e0 (0x2175 bytes) 000028[0] 1765103517.166800 mapwrite 0x12a8d2 thru 0x137ee6 (0xd615 bytes) 000029[1] 1765103517.307471 mapread 0x12835d thru 0x132ecd (0xab71 bytes) 000030[0] 1765103517.373049 write 0x1ace5a thru 0x1b0d15 (0x3ebc bytes) 000032[0] 1765103517.435512 write 0x5d5eff thru 0x5e58fa (0xf9fc bytes) 000033[0] 1765103517.467540 mapwrite 0x137e12 thru 0x13dad3 (0x5cc2 bytes) 000034[1] 1765103517.542044 write 0x2536e6 thru 0x2618be (0xe1d9 bytes) 000037[1] 1765103517.660690 trunc from 0x6ac1e1 to 0x71cea3 (0x70cc3 bytes) 000038[1] 1765103517.704978 mapread 0x20c714 thru 0x211f6d (0x585a bytes) 000039[0] 1765103517.721341 read 0x516118 thru 0x516bd4 (0xabd bytes) 000044[1] 1765103517.840533 mapwrite 0x445f02 thru 0x45001a (0xa119 bytes) 000045[1] 1765103518.032854 trunc from 0x71cea4 to 0x8d6929 (0x1b9a86 bytes) 000048[0] 1765103518.116112 write 0x3741f8 thru 0x37dae5 (0x98ee bytes) 000049[1] 1765103518.179613 trunc from 0x8d692a to 0x845080 (0x918a9 bytes) 000052[1] 1765103518.292717 mapwrite 0x028e6c thru 0x032f79 (0xa10e bytes) 000053[1] 1765103518.378350 write 0x91182e thru 0x918dcc (0x759f bytes) 000056[1] 1765103518.412954 trunc from 0x918dcd to 0x5081ac (0x410c20 bytes) 000057[0] 1765103518.440850 read 0x494847 thru 0x49a4db (0x5c95 bytes) 000059[0] 1765103518.476275 write 0x8ae716 thru 0x8bc6db (0xdfc6 bytes) 000062[0] 1765103518.613432 mapwrite 0x46eeb9 thru 0x473bb2 (0x4cfa bytes) 000065[0] 1765103518.750052 mapread 0x7b3a92 thru 0x7bafdc (0x754b bytes) 000067[0] 1765103518.883556 mapwrite 0x52317f thru 0x53002a (0xceac bytes) 000068[0] 1765103519.080347 read 0x63af13 thru 0x63cee6 (0x1fd4 bytes) 000069[0] 1765103519.119269 write 0x94ff65 thru 0x95c26f (0xc30b bytes) 000073[1] 1765103519.165113 read 0x4670a9 thru 0x47033e (0x9296 bytes) 000074[1] 1765103519.188225 write 0x419e18 thru 0x424a41 (0xac2a bytes) 000076[1] 1765103519.265635 mapread 0x34111f thru 0x344b48 (0x3a2a bytes) 000077[0] 1765103519.322186 read 0x8a4c33 thru 0x8aab54 (0x5f22 bytes) 000078[1] 1765103519.373757 mapread 0x6ec97b thru 0x6f1fa6 (0x562c bytes) 000079[0] 1765103519.409046 mapread 0x082354 thru 0x08b906 (0x95b3 bytes) 000081[0] 1765103519.537230 mapwrite 0x5b7eba thru 0x5bfb12 (0x7c59 bytes) 000085[0] 1765103519.782805 read 0x02e740 thru 0x032b06 (0x43c7 bytes) 000087[1] 1765103519.829400 mapread 0x619be8 thru 0x620f7e (0x7397 bytes) 000091[0] 1765103519.907889 mapread 0x4f9f55 thru 0x504c59 (0xad05 bytes) 000094[0] 1765103519.961266 mapwrite 0x580f7c thru 0x588d49 (0x7dce bytes) 000096[0] 1765103520.058655 mapread 0x1e5eea thru 0x1e818f (0x22a6 bytes) 000098[0] 1765103520.128481 read 0x0f5544 thru 0x104598 (0xf055 bytes) 000099[0] 1765103520.185338 write 0x4005db thru 0x40fcde (0xf704 bytes) 000100[1] 1765103520.241404 trunc from 0x95c270 to 0x457661 (0x504c0e bytes) 000101[0] 1765103520.348759 write 0x15cb0a thru 0x16130f (0x4806 bytes) 000105[1] 1765103520.467408 mapwrite 0x34a293 thru 0x34d877 (0x35e5 bytes) 000109[1] 1765103520.631962 mapread 0x35d4f9 thru 0x368a0e (0xb516 bytes) 000111[1] 1765103520.670498 trunc from 0x457662 to 0x7efca0 (0x39863f bytes) 000112[1] 1765103520.693656 write 0x4f921b thru 0x4fcb7d (0x3963 bytes) 000115[1] 1765103520.705466 read 0x731f1d thru 0x73f0f5 (0xd1d9 bytes) 000116[0] 1765103520.754956 write 0x09e60f thru 0x09fb76 (0x1568 bytes) 000120[1] 1765103520.840311 trunc from 0x7efca1 to 0x235d80 (0x5b9f20 bytes) 000122[0] 1765103520.945968 mapwrite 0x75a895 thru 0x761000 (0x676c bytes) 000123[0] 1765103521.131256 read 0x413f84 thru 0x4224db (0xe558 bytes) 000127[0] 1765103521.167043 mapwrite 0x580ef2 thru 0x587476 (0x6585 bytes) 000131[1] 1765103521.330418 trunc from 0x761001 to 0x9bc59d (0x25b59d bytes) 000132[1] 1765103521.397858 write 0x152770 thru 0x158bc6 (0x6457 bytes) 000135[1] 1765103521.427460 mapread 0x618f8b thru 0x619648 (0x6be bytes) 000136[0] 1765103521.436443 read 0x15137a thru 0x151efc (0xb83 bytes) 000138[1] 1765103521.478662 write 0x03d7fc thru 0x0411ef (0x39f4 bytes) 000139[1] 1765103521.525290 trunc from 0x9bc59e to 0x0d3080 (0x8e951d bytes) 000140[0] 1765103521.584335 mapwrite 0x4daf20 thru 0x4e192c (0x6a0d bytes) 000141[0] 1765103521.703500 read 0x3a5270 thru 0x3a8a9f (0x3830 bytes) 000142[0] 1765103521.708994 mapwrite 0x8ea58c thru 0x8f9371 (0xede6 bytes) 000145[0] 1765103521.946743 read 0x3ceff6 thru 0x3ddd16 (0xed21 bytes) 000148[1] 1765103522.023675 read 0x1659a7 thru 0x17206c (0xc6c6 bytes) 000149[0] 1765103522.081525 mapread 0x82fb29 thru 0x83c739 (0xcc11 bytes) 000150[0] 1765103522.128717 read 0x059ef9 thru 0x06582e (0xb936 bytes) 000152[1] 1765103522.153796 mapread 0x780fdb thru 0x781f1f (0xf45 bytes) 000154[0] 1765103522.187226 mapread 0x5605c2 thru 0x56dc11 (0xd650 bytes) 000155[0] 1765103522.214627 mapread 0x08d5ef thru 0x08fc9c (0x26ae bytes) 000157[0] 1765103522.236844 write 0x567c37 thru 0x571ff2 (0xa3bc bytes) 000159[1] 1765103522.260699 write 0x5f50b5 thru 0x5ffa05 (0xa951 bytes) 000164[0] 1765103522.352948 mapwrite 0x36f925 thru 0x37169e (0x1d7a bytes) 000166[1] 1765103522.504167 mapwrite 0x58f603 thru 0x591dc3 (0x27c1 bytes) 000171[0] 1765103522.672861 mapread 0x47a893 thru 0x47f9cb (0x5139 bytes) 000172[0] 1765103522.722730 mapread 0x727681 thru 0x7299b5 (0x2335 bytes) 000174[0] 1765103522.755751 mapwrite 0x0ce348 thru 0x0cf97f (0x1638 bytes) 000178[1] 1765103522.846386 mapread 0x1be40a thru 0x1c0065 (0x1c5c bytes) 000184[0] 1765103522.949477 trunc from 0x8f9372 to 0x9d498b (0xdb61a bytes) 000186[0] 1765103522.998363 trunc from 0x9d498c to 0x5c2f9a (0x4119f1 bytes) 000187[0] 1765103523.033701 write 0x2d6278 thru 0x2d70ff (0xe88 bytes) 000188[1] 1765103523.045737 write 0x1a87f1 thru 0x1b59bf (0xd1cf bytes) 000190[1] 1765103523.116945 trunc from 0x5c2f9b to 0x50e92a (0xb4670 bytes) 000193[1] 1765103523.194288 trunc from 0x50e92b to 0x86c505 (0x35dbdb bytes) truncating to largest ever: 0x9fa65f 000197[1] 1765103523.258905 trunc from 0x86c506 to 0x9fa65e (0x18e159 bytes) 000199[1] 1765103523.292009 read 0x1c122b thru 0x1cf112 (0xdee8 bytes) 000201[1] 1765103523.351682 read 0x9a77f1 thru 0x9ace22 (0x5632 bytes) 000202[1] 1765103523.375856 write 0x62a846 thru 0x63499e (0xa159 bytes) 000203[1] 1765103523.391110 trunc from 0x9fa65f to 0x6e8621 (0x31203d bytes) 000204[0] 1765103523.467251 write 0x2e7917 thru 0x2f472f (0xce19 bytes) 000206[1] 1765103523.571707 mapread 0x584ecc thru 0x593cb3 (0xede8 bytes) 000207[1] 1765103523.658285 trunc from 0x6e8622 to 0x239a56 (0x4aebcb bytes) 000209[1] 1765103523.749964 mapread 0x0a8a1a thru 0x0b2e00 (0xa3e7 bytes) 000213[0] 1765103523.883518 mapwrite 0x4c86dd thru 0x4cd7ef (0x5113 bytes) 000218[1] 1765103524.122495 mapwrite 0x542f3c thru 0x5516c4 (0xe789 bytes) 000219[0] 1765103524.347144 write 0x8b42db thru 0x8c1764 (0xd48a bytes) 000220[1] 1765103524.393411 trunc from 0x8c1765 to 0x790244 (0x131520 bytes) 000222[0] 1765103524.476446 read 0x006054 thru 0x00fe17 (0x9dc4 bytes) 000223[1] 1765103524.547355 mapread 0x386a41 thru 0x38c6d2 (0x5c92 bytes) 000224[1] 1765103524.574678 mapwrite 0x680bd9 thru 0x68407b (0x34a3 bytes) 000226[0] 1765103524.676689 write 0x298c4f thru 0x29ee1e (0x61d0 bytes) 000228[0] 1765103524.735188 mapwrite 0x6c9370 thru 0x6cd5ba (0x424b bytes) 000229[1] 1765103524.808007 mapwrite 0x841a63 thru 0x84a168 (0x8706 bytes) 000230[1] 1765103524.976455 read 0x490b8a thru 0x496249 (0x56c0 bytes) 000232[1] 1765103525.079412 mapread 0x146e32 thru 0x14c722 (0x58f1 bytes) 000234[0] 1765103525.105832 read 0x2422d3 thru 0x251876 (0xf5a4 bytes) 000235[1] 1765103525.147765 write 0x886f58 thru 0x8953e3 (0xe48c bytes) 000237[0] 1765103525.191617 read 0x53452f thru 0x541219 (0xcceb bytes) 000240[0] 1765103525.242297 read 0x42649d thru 0x435f7d (0xfae1 bytes) 000241[1] 1765103525.279110 trunc from 0x8953e4 to 0x39cbfc (0x4f87e7 bytes) 000244[1] 1765103525.370692 trunc from 0x39cbfd to 0x83f86d (0x4a2c71 bytes) 000249[1] 1765103525.469969 read 0x39a5c8 thru 0x3a18b0 (0x72e9 bytes) 000250[1] 1765103525.495901 mapread 0x480544 thru 0x489a17 (0x94d4 bytes) 000255[1] 1765103525.580911 mapread 0x1dc57e thru 0x1e5710 (0x9193 bytes) 000259[1] 1765103525.678764 trunc from 0x83f86e to 0x332aa0 (0x50cdcd bytes) 000261[0] 1765103525.754430 write 0x1e459a thru 0x1f1293 (0xccfa bytes) 000262[1] 1765103525.833025 trunc from 0x332aa1 to 0x46c9c1 (0x139f21 bytes) 000263[0] 1765103525.879584 write 0x4253c9 thru 0x427cf7 (0x292f bytes) 000269[0] 1765103526.014704 trunc from 0x46c9c2 to 0x86aba7 (0x3fe1e6 bytes) 000271[0] 1765103526.086965 mapwrite 0x2719c7 thru 0x27fc66 (0xe2a0 bytes) 000272[0] 1765103526.159421 mapread 0x6fbc2e thru 0x6fc935 (0xd08 bytes) 000273[1] 1765103526.182286 write 0x75f38a thru 0x764a89 (0x5700 bytes) 000278[1] 1765103526.341682 write 0x288821 thru 0x294a6a (0xc24a bytes) 000279[0] 1765103526.382844 read 0x69045f thru 0x694a24 (0x45c6 bytes) 000282[1] 1765103526.458329 trunc from 0x86aba8 to 0x77de94 (0xecd13 bytes) 000284[0] 1765103526.642822 write 0x8ca408 thru 0x8cbdb3 (0x19ac bytes) 000287[0] 1765103526.719026 mapread 0x5d83f5 thru 0x5de2f6 (0x5f02 bytes) 000289[1] 1765103526.845381 trunc from 0x8cbdb4 to 0x3eaa43 (0x4e1370 bytes) 000290[1] 1765103526.933643 mapread 0x39f73a thru 0x3a5ec6 (0x678d bytes) 000291[0] 1765103526.967601 write 0x5b4a8b thru 0x5bfd32 (0xb2a8 bytes) 000292[1] 1765103526.998395 write 0x144da0 thru 0x145dcf (0x1030 bytes) 000296[0] 1765103527.075333 mapwrite 0x0485d8 thru 0x04b5ec (0x3015 bytes) 000298[0] 1765103527.259212 write 0x9bdc4b thru 0x9c0b22 (0x2ed8 bytes) 000304[1] 1765103527.341419 mapwrite 0x77fc49 thru 0x781cdf (0x2097 bytes) 000306[1] 1765103527.646199 trunc from 0x9c0b23 to 0x4b5710 (0x50b412 bytes) 000308[0] 1765103527.728987 trunc from 0x4b5711 to 0x4e56fb (0x2ffeb bytes) 000313[1] 1765103527.878382 write 0x02409d thru 0x0253a6 (0x130a bytes) 000314[0] 1765103527.927710 write 0x185dba thru 0x18ea7d (0x8cc4 bytes) 000320[0] 1765103528.035491 mapwrite 0x028043 thru 0x03518a (0xd148 bytes) 000325[0] 1765103528.205871 read 0x337dc1 thru 0x343db6 (0xbff6 bytes) 000326[0] 1765103528.251365 read 0x2f19a2 thru 0x2f879c (0x6dfb bytes) 000327[1] 1765103528.287210 mapwrite 0x666d64 thru 0x67158a (0xa827 bytes) 000331[1] 1765103528.476653 write 0x16844a thru 0x177255 (0xee0c bytes) 000332[0] 1765103528.488773 mapwrite 0x813a21 thru 0x81e5ba (0xab9a bytes) 000335[0] 1765103528.728133 mapwrite 0x6485b3 thru 0x648baf (0x5fd bytes) 000336[1] 1765103528.802214 trunc from 0x81e5bb to 0x395f93 (0x488627 bytes) 000337[0] 1765103528.894475 mapread 0x13e773 thru 0x148a24 (0xa2b2 bytes) 000338[0] 1765103528.940699 mapwrite 0x485178 thru 0x485529 (0x3b2 bytes) 000340[0] 1765103529.095555 trunc from 0x48552a to 0x9c6c2d (0x541704 bytes) 000341[0] 1765103529.117655 write 0x1acf3c thru 0x1b31c8 (0x628d bytes) 000343[0] 1765103529.154477 write 0x1ad428 thru 0x1b227f (0x4e58 bytes) 000350[1] 1765103529.257480 read 0x42ee03 thru 0x4397e8 (0xa9e6 bytes) 000354[0] 1765103529.326191 mapwrite 0x47ce80 thru 0x48aabe (0xdc3f bytes) 000355[0] 1765103529.461238 mapwrite 0x8b0597 thru 0x8bd5e4 (0xd04e bytes) 000358[1] 1765103529.585222 mapread 0x51c371 thru 0x52197a (0x560a bytes) 000359[0] 1765103529.640133 write 0x8a0582 thru 0x8a888b (0x830a bytes) 000360[0] 1765103529.689201 trunc from 0x9c6c2e to 0x4b3b65 (0x5130c8 bytes) 000361[1] 1765103529.793841 mapread 0x356fb4 thru 0x361050 (0xa09d bytes) 000364[0] 1765103529.890312 mapread 0x3a7ede thru 0x3aa529 (0x264c bytes) 000365[0] 1765103529.922381 read 0x30dbf2 thru 0x31bd1f (0xe12e bytes) 000366[1] 1765103529.937528 write 0x612e66 thru 0x618ef1 (0x608c bytes) 000372[0] 1765103530.053909 read 0x0d6a9e thru 0x0db9ff (0x4f62 bytes) 000380[1] 1765103530.138346 mapread 0x0d855b thru 0x0e399c (0xb442 bytes) 000381[0] 1765103530.161965 read 0x204abb thru 0x20f160 (0xa6a6 bytes) 000383[1] 1765103530.195708 write 0x1ea7dd thru 0x1f913e (0xe962 bytes) 000385[1] 1765103530.269655 mapread 0x130987 thru 0x13347c (0x2af6 bytes) 000387[0] 1765103530.281003 mapread 0x4ba581 thru 0x4c6d0a (0xc78a bytes) 000389[0] 1765103530.362863 read 0x0534cc thru 0x060ece (0xda03 bytes) 000393[1] 1765103530.411798 trunc from 0x618ef2 to 0x76a79a (0x1518a9 bytes) 000394[0] 1765103530.489657 trunc from 0x76a79b to 0x2bdafb (0x4acc9f bytes) 000395[1] 1765103530.569581 trunc from 0x2bdafc to 0x88924f (0x5cb754 bytes) 000397[1] 1765103530.618003 trunc from 0x889250 to 0x7ec944 (0x9c90b bytes) 000398[1] 1765103530.631921 mapread 0x7b18fe thru 0x7b8a69 (0x716c bytes) 000405[1] 1765103530.793134 trunc from 0x7ec945 to 0x608e0f (0x1e3b35 bytes) 000406[0] 1765103530.844443 mapread 0x1bb1c1 thru 0x1bf9bb (0x47fb bytes) 000407[1] 1765103530.898118 mapwrite 0x54f215 thru 0x550b69 (0x1955 bytes) 000408[0] 1765103530.983566 trunc from 0x608e10 to 0x6a9078 (0xa0269 bytes) 000410[1] 1765103531.112583 mapread 0x2eaa09 thru 0x2f86f8 (0xdcf0 bytes) 000413[0] 1765103531.209881 mapwrite 0x4703ce thru 0x47446b (0x409e bytes) 000414[0] 1765103531.349766 read 0x03e3fc thru 0x043815 (0x541a bytes) 000416[0] 1765103531.390366 read 0x56d8a3 thru 0x576c68 (0x93c6 bytes) 000418[1] 1765103531.424939 mapread 0x49a1dc thru 0x4a1e42 (0x7c67 bytes) 000419[1] 1765103531.480842 write 0x5de6e7 thru 0x5e9196 (0xaab0 bytes) 000420[1] 1765103531.519430 write 0x8cc078 thru 0x8d04f0 (0x4479 bytes) 000423[0] 1765103531.596501 write 0x7d77b1 thru 0x7e321f (0xba6f bytes) 000424[0] 1765103531.633682 mapwrite 0x07446b thru 0x0746a4 (0x23a bytes) 000425[1] 1765103531.736578 write 0x6ac529 thru 0x6afc23 (0x36fb bytes) 000426[0] 1765103531.808730 read 0x0fe828 thru 0x10139c (0x2b75 bytes) 000427[1] 1765103531.846704 mapwrite 0x8e1684 thru 0x8ee0fa (0xca77 bytes) 000428[0] 1765103531.980514 read 0x7e8e40 thru 0x7f5a9c (0xcc5d bytes) 000429[1] 1765103532.031429 trunc from 0x8ee0fb to 0x5eacc8 (0x303432 bytes) 000432[0] 1765103532.087299 write 0x724344 thru 0x730d9f (0xca5c bytes) 000434[1] 1765103532.183832 trunc from 0x730da0 to 0x382667 (0x3ae738 bytes) 000438[1] 1765103532.398440 write 0x45c2cf thru 0x4672be (0xaff0 bytes) 000442[1] 1765103532.443209 mapread 0x0d359c thru 0x0dd9b8 (0xa41d bytes) 000443[0] 1765103532.463385 mapwrite 0x228417 thru 0x234431 (0xc01b bytes) 000444[0] 1765103532.563730 mapwrite 0x784178 thru 0x79165b (0xd4e4 bytes) 000446[1] 1765103532.776510 write 0x255fe9 thru 0x25d767 (0x777f bytes) 000448[0] 1765103532.825293 read 0x5c5b6e thru 0x5c66d3 (0xb66 bytes) 000450[1] 1765103532.885398 trunc from 0x79165c to 0x32ad5c (0x4668ff bytes) 000451[1] 1765103532.959519 mapwrite 0x1e6f91 thru 0x1ee0dc (0x714c bytes) 000453[0] 1765103533.073915 read 0x31790d thru 0x31958a (0x1c7e bytes) 000455[1] 1765103533.228720 mapwrite 0x703093 thru 0x706bb5 (0x3b23 bytes) 000457[0] 1765103533.368732 mapwrite 0x3a65d3 thru 0x3afed2 (0x9900 bytes) 000458[1] 1765103533.507556 write 0x8d0bfb thru 0x8d7a98 (0x6e9e bytes) 000459[0] 1765103533.549959 mapwrite 0x647461 thru 0x64b190 (0x3d30 bytes) 000460[0] 1765103533.707722 mapread 0x09c770 thru 0x0ac608 (0xfe99 bytes) 000461[1] 1765103533.865920 write 0x08bce1 thru 0x09584b (0x9b6b bytes) 000462[0] 1765103533.957456 trunc from 0x8d7a99 to 0x9b4951 (0xdceb9 bytes) 000467[1] 1765103534.125123 mapread 0x9aee1c thru 0x9b4951 (0x5b36 bytes) 000470[0] 1765103534.268910 trunc from 0x9b4952 to 0x992d07 (0x21c4a bytes) 000472[1] 1765103534.332366 write 0x6531e8 thru 0x659ccf (0x6ae8 bytes) 000476[1] 1765103534.433496 write 0x57fc2c thru 0x589026 (0x93fb bytes) 000477[0] 1765103534.463280 read 0x0bf8d3 thru 0x0c698c (0x70ba bytes) 000479[1] 1765103534.500917 read 0x1b3e28 thru 0x1bda3b (0x9c14 bytes) 000481[0] 1765103534.530328 read 0x5b40ce thru 0x5bfb05 (0xba38 bytes) 000485[1] 1765103534.567102 trunc from 0x992d08 to 0x205573 (0x78d794 bytes) 000486[1] 1765103534.642151 mapwrite 0x19d7b9 thru 0x1a47c0 (0x7008 bytes) 000487[0] 1765103534.746103 read 0x0ad667 thru 0x0b1835 (0x41cf bytes) 000489[1] 1765103534.888108 mapread 0x17a08d thru 0x18415a (0xa0ce bytes) 000492[0] 1765103534.920151 write 0x76f18f thru 0x77e05e (0xeed0 bytes) 000493[0] 1765103534.937402 write 0x4dba82 thru 0x4e6bf1 (0xb170 bytes) 000494[1] 1765103534.969401 trunc from 0x77e05f to 0x7fa928 (0x7c8ca bytes) 000495[1] 1765103535.026377 write 0x1f125a thru 0x1f12b4 (0x5b bytes) 000497[1] 1765103535.081117 mapwrite 0x595293 thru 0x59f151 (0x9ebf bytes) 000498[0] 1765103535.153156 mapwrite 0x17a6da thru 0x180db0 (0x66d7 bytes) 000499[0] 1765103535.349722 write 0x361b80 thru 0x36c20a (0xa68b bytes) 000500[0] 1765103535.363728 read 0x491ba3 thru 0x4961b9 (0x4617 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3535 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x57c7ec 000003[0] 1765103536.044714 trunc from 00000000 to 0x57c7eb (0x57c7ec bytes) 000005[0] 1765103536.142348 write 0x3a7000 thru 0x3a89ef (0x19f0 bytes) 000009[0] 1765103536.208000 trunc from 0x57c7ec to 0x11c897 (0x45ff54 bytes) 000010[0] 1765103536.263629 write 0x8c2000 thru 0x8cebb6 (0xcbb7 bytes) 000014[1] 1765103536.304747 trunc from 0x8cebb7 to 0x574e6a (0x359d4c bytes) 000015[1] 1765103536.349528 mapwrite 0x64d000 thru 0x657d83 (0xad84 bytes) 000017[0] 1765103536.434329 write 0x6f4000 thru 0x7030af (0xf0b0 bytes) truncating to largest ever: 0x60480b 000018[1] 1765103536.468405 trunc from 0x7030b0 to 0x60480a (0xfe8a5 bytes) 000020[0] 1765103536.534679 write 0x89f000 thru 0x8a6594 (0x7595 bytes) 000026[0] 1765103536.623684 read 0x771000 thru 0x779d38 (0x8d39 bytes) 000027[0] 1765103536.660684 mapread 0x5de000 thru 0x5e98ff (0xb900 bytes) 000028[1] 1765103536.729757 write 0x58f000 thru 0x59d321 (0xe322 bytes) 000030[0] 1765103536.817822 trunc from 0x8a6595 to 0x31413e (0x592456 bytes) 000031[1] 1765103536.903307 trunc from 0x31413f to 0x057d11 (0x2bc42d bytes) 000032[0] 1765103536.933784 mapread 0x03a000 thru 0x03b714 (0x1715 bytes) 000033[0] 1765103536.984518 read 0x03d000 thru 0x04b0cb (0xe0cc bytes) 000034[0] 1765103537.011451 read 0x047000 thru 0x049f37 (0x2f38 bytes) truncating to largest ever: 0x859a17 000036[1] 1765103537.019898 trunc from 0x057d12 to 0x859a16 (0x801d05 bytes) 000041[0] 1765103537.112414 mapread 0x217000 thru 0x217d54 (0xd55 bytes) 000043[0] 1765103537.201355 read 0x216000 thru 0x222746 (0xc747 bytes) 000044[0] 1765103537.201866 mapread 0x712000 thru 0x7182a1 (0x62a2 bytes) 000045[1] 1765103537.250615 write 0x723000 thru 0x728e94 (0x5e95 bytes) 000047[0] 1765103537.305919 mapwrite 0x8ba000 thru 0x8be62c (0x462d bytes) 000048[0] 1765103537.472372 write 0x35b000 thru 0x35fb19 (0x4b1a bytes) 000049[0] 1765103537.487205 read 0x639000 thru 0x6480be (0xf0bf bytes) 000050[0] 1765103537.531951 write 0x717000 thru 0x7247b3 (0xd7b4 bytes) 000051[0] 1765103537.542096 read 0x16d000 thru 0x17459f (0x75a0 bytes) 000053[1] 1765103537.602722 mapwrite 0x3b4000 thru 0x3bbbb7 (0x7bb8 bytes) truncating to largest ever: 0x8bb434 000056[0] 1765103537.813179 trunc from 0x8be62d to 0x8bb433 (0x31f9 bytes) 000057[1] 1765103537.875402 mapwrite 0x721000 thru 0x727444 (0x6445 bytes) 000058[1] 1765103537.991783 trunc from 0x8bb434 to 0x7086e7 (0x1b2d4c bytes) 000062[0] 1765103538.048238 mapread 0x261000 thru 0x26c4b6 (0xb4b7 bytes) 000064[1] 1765103538.129514 mapread 0x488000 thru 0x48d014 (0x5015 bytes) 000072[1] 1765103538.161969 mapwrite 0x7d9000 thru 0x7e55ed (0xc5ee bytes) 000073[0] 1765103538.274300 write 0x050000 thru 0x05e4bf (0xe4c0 bytes) 000077[0] 1765103538.360014 mapwrite 0x5b7000 thru 0x5c5a0e (0xea0f bytes) 000080[0] 1765103538.473282 read 0x6c0000 thru 0x6c69d4 (0x69d5 bytes) 000081[0] 1765103538.500932 read 0x76e000 thru 0x76ffd1 (0x1fd2 bytes) 000084[0] 1765103538.536695 trunc from 0x7e55ee to 0x6d2ebe (0x11272f bytes) 000085[1] 1765103538.579395 read 0x05c000 thru 0x06157a (0x557b bytes) 000088[0] 1765103538.608519 mapread 0x176000 thru 0x18029c (0xa29d bytes) 000089[0] 1765103538.628213 read 0x15b000 thru 0x165b97 (0xab98 bytes) 000090[1] 1765103538.637374 write 0x7af000 thru 0x7b6342 (0x7343 bytes) 000094[0] 1765103538.653120 write 0x983000 thru 0x98dba4 (0xaba5 bytes) 000097[0] 1765103538.705850 write 0x59a000 thru 0x5a3f71 (0x9f72 bytes) 000098[0] 1765103538.712353 mapwrite 0x896000 thru 0x899055 (0x3056 bytes) 000101[0] 1765103538.800868 mapread 0x1f7000 thru 0x1ff876 (0x8877 bytes) 000103[0] 1765103538.833684 mapwrite 0x9eb000 thru 0x9fa6c3 (0xf6c4 bytes) 000105[1] 1765103538.966121 write 0x963000 thru 0x96f620 (0xc621 bytes) 000111[0] 1765103539.020985 mapwrite 0x785000 thru 0x788002 (0x3003 bytes) 000112[1] 1765103539.125319 mapwrite 0x860000 thru 0x866959 (0x695a bytes) 000114[1] 1765103539.211453 mapwrite 0x1d5000 thru 0x1e164a (0xc64b bytes) 000116[0] 1765103539.358925 read 0x643000 thru 0x644904 (0x1905 bytes) 000117[1] 1765103539.374787 write 0x964000 thru 0x96e55f (0xa560 bytes) 000119[1] 1765103539.390413 write 0x464000 thru 0x46bc7b (0x7c7c bytes) 000122[1] 1765103539.426713 write 0x232000 thru 0x2344e7 (0x24e8 bytes) 000123[1] 1765103539.457596 read 0x1b3000 thru 0x1bdcc4 (0xacc5 bytes) 000125[1] 1765103539.506979 trunc from 0x9fa6c4 to 0x176684 (0x88403f bytes) 000127[0] 1765103539.527517 mapread 0x163000 thru 0x16cf1f (0x9f20 bytes) 000129[1] 1765103539.573875 read 0x115000 thru 0x123444 (0xe445 bytes) 000130[0] 1765103539.610142 trunc from 0x176685 to 0x7dae01 (0x66477d bytes) 000133[1] 1765103539.683100 mapread 0x68c000 thru 0x68c486 (0x487 bytes) 000143[1] 1765103539.771108 trunc from 0x7dae02 to 0x7ddf3d (0x313c bytes) 000144[1] 1765103539.825221 trunc from 0x7ddf3e to 0x58d0f5 (0x250e48 bytes) 000146[0] 1765103539.885559 mapread 0x588000 thru 0x58c98e (0x498f bytes) 000151[1] 1765103539.958700 mapwrite 0x10b000 thru 0x113c2a (0x8c2b bytes) 000153[1] 1765103540.163796 mapread 0x0b0000 thru 0x0b3ad0 (0x3ad1 bytes) 000154[1] 1765103540.217335 mapwrite 0x8a0000 thru 0x8a97a0 (0x97a1 bytes) 000155[0] 1765103540.380623 mapwrite 0x5ee000 thru 0x5f8a38 (0xaa39 bytes) 000159[1] 1765103540.482186 write 0x780000 thru 0x786b83 (0x6b84 bytes) 000160[0] 1765103540.509039 write 0x1bd000 thru 0x1c65f0 (0x95f1 bytes) 000163[1] 1765103540.568831 mapwrite 0x0be000 thru 0x0c22e2 (0x42e3 bytes) 000167[0] 1765103540.689283 mapwrite 0x717000 thru 0x71e67c (0x767d bytes) 000168[0] 1765103540.795321 mapwrite 0x3a0000 thru 0x3ad381 (0xd382 bytes) 000173[0] 1765103540.908535 trunc from 0x8a97a1 to 0x5bfed6 (0x2e98ca bytes) 000175[1] 1765103540.933303 mapwrite 0x2dc000 thru 0x2e5814 (0x9815 bytes) 000176[0] 1765103541.138788 mapread 0x184000 thru 0x186b91 (0x2b92 bytes) 000177[0] 1765103541.182786 mapwrite 0x7d4000 thru 0x7dcb20 (0x8b21 bytes) 000179[0] 1765103541.325822 mapwrite 0x8d7000 thru 0x8dab7d (0x3b7e bytes) 000180[0] 1765103541.434798 read 0x3f6000 thru 0x40443b (0xe43c bytes) 000181[0] 1765103541.455555 write 0x882000 thru 0x88750f (0x5510 bytes) 000182[0] 1765103541.487442 mapread 0x304000 thru 0x306c65 (0x2c66 bytes) 000188[0] 1765103541.559601 mapwrite 0x0b9000 thru 0x0bb7f2 (0x27f3 bytes) 000189[0] 1765103541.660864 mapwrite 0x764000 thru 0x76afc5 (0x6fc6 bytes) 000190[0] 1765103541.718543 mapwrite 0x478000 thru 0x478019 (0x1a bytes) 000199[1] 1765103541.826327 trunc from 0x8dab7e to 0x86bab2 (0x6f0cb bytes) 000206[1] 1765103541.913432 write 0x43b000 thru 0x445190 (0xa191 bytes) 000207[0] 1765103541.937531 trunc from 0x86bab3 to 0x37cd95 (0x4eed1d bytes) 000208[0] 1765103542.007846 read 0x2b4000 thru 0x2b7944 (0x3945 bytes) 000209[1] 1765103542.097636 mapwrite 0x151000 thru 0x158ee3 (0x7ee4 bytes) 000210[0] 1765103542.271246 mapwrite 0x8df000 thru 0x8e47a1 (0x57a2 bytes) 000211[0] 1765103542.502523 write 0x2bd000 thru 0x2ca538 (0xd539 bytes) 000214[1] 1765103542.567042 read 0x38f000 thru 0x39e393 (0xf394 bytes) 000216[0] 1765103542.617121 mapwrite 0x134000 thru 0x14095c (0xc95d bytes) 000217[0] 1765103542.731192 write 0x236000 thru 0x23a5a7 (0x45a8 bytes) 000219[0] 1765103542.761351 mapwrite 0x9b1000 thru 0x9bda0d (0xca0e bytes) 000220[1] 1765103542.845757 read 0x8d9000 thru 0x8d9e8e (0xe8f bytes) 000222[1] 1765103542.872579 trunc from 0x9bda0e to 0x7b8992 (0x20507b bytes) 000224[1] 1765103542.916424 read 0x5ef000 thru 0x5fba43 (0xca44 bytes) 000225[0] 1765103542.943333 trunc from 0x7b8993 to 0x195a30 (0x622f62 bytes) 000226[1] 1765103542.983049 mapwrite 0x65a000 thru 0x660765 (0x6766 bytes) 000227[0] 1765103543.086512 mapwrite 0x095000 thru 0x0a0797 (0xb798 bytes) 000228[0] 1765103543.207434 mapread 0x5f5000 thru 0x604aba (0xfabb bytes) 000229[0] 1765103543.291705 mapread 0x340000 thru 0x34f6e4 (0xf6e5 bytes) truncating to largest ever: 0x955495 000232[1] 1765103543.398428 trunc from 0x660766 to 0x955494 (0x2f4d2f bytes) 000236[1] 1765103543.467598 write 0x1d3000 thru 0x1da4fe (0x74ff bytes) 000240[1] 1765103543.519630 mapwrite 0x417000 thru 0x421210 (0xa211 bytes) 000241[0] 1765103543.617884 mapwrite 0x8e5000 thru 0x8f25a6 (0xd5a7 bytes) 000242[0] 1765103543.806905 mapwrite 0x2cf000 thru 0x2de1dd (0xf1de bytes) 000243[1] 1765103544.023952 write 0x9a5000 thru 0x9afe50 (0xae51 bytes) 000246[0] 1765103544.065883 mapread 0x26f000 thru 0x27930f (0xa310 bytes) 000247[0] 1765103544.184898 mapread 0x3f1000 thru 0x3f22d2 (0x12d3 bytes) 000248[1] 1765103544.195528 read 0x092000 thru 0x09bd94 (0x9d95 bytes) 000249[0] 1765103544.202307 mapwrite 0x677000 thru 0x67ddbd (0x6dbe bytes) 000250[1] 1765103544.375623 mapread 0x294000 thru 0x2a219a (0xe19b bytes) 000252[1] 1765103544.453735 trunc from 0x9afe51 to 0x6a0d79 (0x30f0d7 bytes) 000253[0] 1765103544.513138 write 0x26d000 thru 0x26f7ea (0x27eb bytes) 000254[0] 1765103544.548838 mapwrite 0x17a000 thru 0x17dcdb (0x3cdc bytes) 000255[1] 1765103544.606566 read 0x166000 thru 0x171a09 (0xba0a bytes) 000256[0] 1765103544.629925 write 0x710000 thru 0x71eb50 (0xeb51 bytes) 000258[0] 1765103544.650888 read 0x393000 thru 0x39ae7e (0x7e7f bytes) 000260[0] 1765103544.681301 mapread 0x580000 thru 0x58d684 (0xd685 bytes) 000261[1] 1765103544.744391 read 0x6d2000 thru 0x6e0468 (0xe469 bytes) 000262[0] 1765103544.799400 read 0x150000 thru 0x1555fa (0x55fb bytes) 000263[1] 1765103544.807399 read 0x4b6000 thru 0x4bac62 (0x4c63 bytes) 000265[0] 1765103544.820539 write 0x305000 thru 0x314d0d (0xfd0e bytes) 000268[0] 1765103544.872142 write 0x2fd000 thru 0x306205 (0x9206 bytes) 000271[0] 1765103544.889787 mapwrite 0x09e000 thru 0x0a0343 (0x2344 bytes) 000272[0] 1765103544.986036 mapread 0x3e5000 thru 0x3f3043 (0xe044 bytes) 000273[1] 1765103545.015428 trunc from 0x71eb51 to 0x8b1036 (0x1924e6 bytes) 000274[0] 1765103545.064946 mapwrite 0x7a3000 thru 0x7a5ce8 (0x2ce9 bytes) 000276[0] 1765103545.171499 mapread 0x017000 thru 0x01dd77 (0x6d78 bytes) 000277[1] 1765103545.207859 mapwrite 0x5a7000 thru 0x5b5c7e (0xec7f bytes) 000278[1] 1765103545.314352 trunc from 0x8b1037 to 0x2b73d4 (0x5f9c62 bytes) 000281[0] 1765103545.378965 read 0x1f1000 thru 0x1fe6c3 (0xd6c4 bytes) 000282[1] 1765103545.444443 trunc from 0x2b73d5 to 0x28060d (0x36dc7 bytes) 000283[0] 1765103545.501250 mapwrite 0x751000 thru 0x7533f6 (0x23f7 bytes) 000285[0] 1765103545.707548 mapread 0x4f6000 thru 0x4fe01d (0x801e bytes) 000288[0] 1765103545.726255 mapwrite 0x296000 thru 0x29be78 (0x5e79 bytes) 000291[1] 1765103545.822092 mapread 0x5c0000 thru 0x5c29f3 (0x29f4 bytes) 000292[0] 1765103545.865833 write 0x384000 thru 0x39085d (0xc85e bytes) 000293[1] 1765103545.887366 read 0x72b000 thru 0x730ec1 (0x5ec2 bytes) 000296[1] 1765103545.926622 trunc from 0x7533f7 to 0x0178b8 (0x73bb3e bytes) 000298[1] 1765103545.971470 read 0x016000 thru 0x01710e (0x110f bytes) 000302[0] 1765103545.994289 trunc from 0x0178b9 to 0x809f54 (0x7f269c bytes) 000303[1] 1765103546.058335 read 0x14c000 thru 0x14ee00 (0x2e01 bytes) 000306[1] 1765103546.088995 write 0x0ce000 thru 0x0dba82 (0xda83 bytes) 000308[1] 1765103546.122184 mapread 0x7d4000 thru 0x7e3c46 (0xfc47 bytes) 000309[0] 1765103546.183354 trunc from 0x809f55 to 0x40094d (0x409607 bytes) 000311[1] 1765103546.233660 trunc from 0x40094e to 0x5fb60d (0x1facc0 bytes) 000316[1] 1765103546.351355 write 0x060000 thru 0x061f22 (0x1f23 bytes) 000317[1] 1765103546.359039 read 0x187000 thru 0x18e9ed (0x79ee bytes) 000318[0] 1765103546.370920 mapwrite 0x796000 thru 0x7a4282 (0xe283 bytes) 000319[1] 1765103546.503508 write 0x065000 thru 0x066087 (0x1088 bytes) 000323[1] 1765103546.559599 mapread 0x66f000 thru 0x67d7a1 (0xe7a2 bytes) 000324[1] 1765103546.585480 trunc from 0x7a4283 to 0x155852 (0x64ea30 bytes) 000326[0] 1765103546.627049 read 0x03b000 thru 0x03e55d (0x355e bytes) 000328[1] 1765103546.681238 write 0x1c8000 thru 0x1d1209 (0x920a bytes) 000330[0] 1765103546.752594 read 0x14d000 thru 0x15123a (0x423b bytes) 000332[0] 1765103546.845198 mapwrite 0x96d000 thru 0x9793b8 (0xc3b9 bytes) 000333[1] 1765103547.092020 trunc from 0x9793b9 to 0x27ac2a (0x6fe78e bytes) 000335[1] 1765103547.192015 mapread 0x21d000 thru 0x22295a (0x595b bytes) 000336[1] 1765103547.214492 mapwrite 0x693000 thru 0x6a1e29 (0xee2a bytes) 000341[1] 1765103547.388216 write 0x83a000 thru 0x845044 (0xb045 bytes) 000342[1] 1765103547.415458 write 0x06d000 thru 0x079c75 (0xcc76 bytes) 000343[0] 1765103547.448045 trunc from 0x845045 to 0x7ba15b (0x8aee9 bytes) 000349[1] 1765103547.547680 read 0x743000 thru 0x7445cc (0x15cd bytes) 000351[1] 1765103547.578839 write 0x8ff000 thru 0x908eca (0x9ecb bytes) 000352[0] 1765103547.606822 read 0x050000 thru 0x05d075 (0xd076 bytes) 000355[0] 1765103547.641587 write 0x79a000 thru 0x79b159 (0x115a bytes) 000357[1] 1765103547.663256 read 0x386000 thru 0x38b677 (0x5678 bytes) 000358[1] 1765103547.694851 trunc from 0x908ecb to 0x26c0a5 (0x69ce25 bytes) 000359[1] 1765103547.744346 write 0x9b7000 thru 0x9c549c (0xe49d bytes) 000360[0] 1765103547.757349 write 0x21c000 thru 0x227b4f (0xbb50 bytes) 000361[1] 1765103547.800842 mapwrite 0x61e000 thru 0x62ce8b (0xee8c bytes) 000363[1] 1765103547.962576 trunc from 0x9c549d to 0x3d1ce6 (0x5f37b6 bytes) 000366[1] 1765103548.019471 read 0x1e9000 thru 0x1f60ae (0xd0af bytes) 000368[0] 1765103548.096999 mapread 0x1a9000 thru 0x1b8a74 (0xfa75 bytes) 000370[1] 1765103548.147196 mapread 0x001000 thru 0x00a67e (0x967f bytes) 000371[1] 1765103548.229990 mapwrite 0x245000 thru 0x24f152 (0xa153 bytes) 000374[1] 1765103548.438021 mapwrite 0x020000 thru 0x025add (0x5ade bytes) 000375[1] 1765103548.569665 mapwrite 0x7de000 thru 0x7de9c8 (0x9c9 bytes) 000376[1] 1765103548.768091 mapread 0x133000 thru 0x13b6d8 (0x86d9 bytes) 000377[0] 1765103548.826432 mapwrite 0x5b8000 thru 0x5bceec (0x4eed bytes) 000378[0] 1765103548.968940 read 0x04c000 thru 0x050eb2 (0x4eb3 bytes) 000379[0] 1765103548.979958 mapread 0x5b9000 thru 0x5c6834 (0xd835 bytes) 000382[0] 1765103549.033203 mapread 0x7a6000 thru 0x7b042b (0xa42c bytes) 000383[1] 1765103549.073221 mapread 0x01c000 thru 0x01d0b2 (0x10b3 bytes) 000386[1] 1765103549.108309 write 0x2f4000 thru 0x2fbbc1 (0x7bc2 bytes) 000387[0] 1765103549.125260 write 0x6fc000 thru 0x706a8f (0xaa90 bytes) 000391[1] 1765103549.200917 read 0x01a000 thru 0x01bf25 (0x1f26 bytes) 000395[1] 1765103549.240363 trunc from 0x7de9c9 to 0x6a1756 (0x13d272 bytes) 000399[0] 1765103549.270446 write 0x614000 thru 0x619b47 (0x5b48 bytes) 000400[0] 1765103549.312842 read 0x3fe000 thru 0x3feee1 (0xee2 bytes) 000401[0] 1765103549.345614 mapread 0x3e9000 thru 0x3ed588 (0x4589 bytes) 000402[1] 1765103549.375567 write 0x459000 thru 0x4680ef (0xf0f0 bytes) 000404[0] 1765103549.414673 trunc from 0x6a1757 to 0x8c70a4 (0x22594e bytes) 000405[0] 1765103549.499418 mapread 0x15f000 thru 0x16a550 (0xb551 bytes) 000407[1] 1765103549.568448 mapwrite 0x725000 thru 0x733473 (0xe474 bytes) 000414[1] 1765103549.739042 mapwrite 0x459000 thru 0x4596b2 (0x6b3 bytes) 000417[0] 1765103549.824948 mapwrite 0x62c000 thru 0x63b9e7 (0xf9e8 bytes) 000420[0] 1765103549.973226 write 0x1db000 thru 0x1db2d3 (0x2d4 bytes) 000423[0] 1765103550.026638 mapwrite 0x70b000 thru 0x711523 (0x6524 bytes) 000424[1] 1765103550.090055 mapread 0x732000 thru 0x73b38d (0x938e bytes) 000425[0] 1765103550.126747 read 0x530000 thru 0x532024 (0x2025 bytes) 000426[1] 1765103550.141558 mapwrite 0x14b000 thru 0x152fa7 (0x7fa8 bytes) 000428[1] 1765103550.226564 mapread 0x433000 thru 0x43cc17 (0x9c18 bytes) 000429[1] 1765103550.247512 trunc from 0x8c70a5 to 0x4ccbef (0x3fa4b5 bytes) 000432[1] 1765103550.325252 mapread 0x35e000 thru 0x368e8f (0xae90 bytes) 000433[0] 1765103550.366944 read 0x4a9000 thru 0x4b270d (0x970e bytes) 000440[0] 1765103550.412394 mapwrite 0x0eb000 thru 0x0ee9f8 (0x39f9 bytes) 000443[1] 1765103550.516991 mapread 0x475000 thru 0x47adf3 (0x5df4 bytes) 000444[1] 1765103550.579495 write 0x83a000 thru 0x842a69 (0x8a6a bytes) 000446[0] 1765103550.622390 mapwrite 0x29d000 thru 0x29d218 (0x219 bytes) 000448[0] 1765103550.718926 write 0x9ec000 thru 0x9fbff7 (0xfff8 bytes) 000449[1] 1765103550.738133 mapwrite 0x972000 thru 0x9745a3 (0x25a4 bytes) 000450[1] 1765103550.846421 mapread 0x08b000 thru 0x090c32 (0x5c33 bytes) 000451[1] 1765103550.892503 write 0x84d000 thru 0x8539e4 (0x69e5 bytes) 000453[0] 1765103550.926284 read 0x383000 thru 0x389aa9 (0x6aaa bytes) 000455[1] 1765103550.954268 mapread 0x076000 thru 0x076fa3 (0xfa4 bytes) 000464[1] 1765103550.962287 mapread 0x75d000 thru 0x766a62 (0x9a63 bytes) 000468[0] 1765103550.982638 read 0x231000 thru 0x23fdd3 (0xedd4 bytes) 000470[0] 1765103550.990989 read 0x1ba000 thru 0x1c671a (0xc71b bytes) 000471[0] 1765103551.000348 read 0x1be000 thru 0x1c6488 (0x8489 bytes) 000472[1] 1765103551.001000 write 0x84c000 thru 0x8569db (0xa9dc bytes) 000473[1] 1765103551.021039 mapwrite 0x762000 thru 0x768dbf (0x6dc0 bytes) 000475[0] 1765103551.090555 mapread 0x929000 thru 0x930436 (0x7437 bytes) 000476[1] 1765103551.146019 read 0x792000 thru 0x799e9c (0x7e9d bytes) 000478[1] 1765103551.163001 trunc from 0x9fbff8 to 0x90e3b4 (0xedc43 bytes) 000481[1] 1765103551.235347 write 0x09f000 thru 0x09f1ea (0x1eb bytes) 000482[0] 1765103551.270564 mapread 0x0c4000 thru 0x0c527f (0x1280 bytes) 000483[1] 1765103551.328424 mapwrite 0x28e000 thru 0x295e62 (0x7e63 bytes) 000485[0] 1765103551.445288 mapwrite 0x42b000 thru 0x42f505 (0x4506 bytes) 000486[0] 1765103551.559052 mapwrite 0x4f2000 thru 0x501905 (0xf906 bytes) 000489[0] 1765103551.670709 write 0x943000 thru 0x94c5bf (0x95c0 bytes) 000490[1] 1765103551.690186 write 0x833000 thru 0x842db7 (0xfdb8 bytes) 000491[1] 1765103551.730804 mapread 0x5f0000 thru 0x5f79a8 (0x79a9 bytes) 000493[1] 1765103551.749698 mapwrite 0x7c4000 thru 0x7c695e (0x295f bytes) 000495[1] 1765103551.820931 mapwrite 0x950000 thru 0x95e3b0 (0xe3b1 bytes) 000498[0] 1765103552.003463 write 0x258000 thru 0x25e1de (0x61df bytes) 000500[0] 1765103552.048764 trunc from 0x95e3b1 to 0x1461cc (0x8181e4 bytes) All operations completed A-OK! PASS 16a (44s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 05:32:35 (1765103555) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.284564 s, 29.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.268715 s, 31.2 MB/s Chance of close/open is 1 in 50 Seed set to 3557 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002[1] 1765103558.405732 mapwrite 0x3aed5a thru 0x3aff0f (0x11b6 bytes) 000003[0] 1765103558.521858 write 0x090421 thru 0x091259 (0xe39 bytes) 000004[0] 1765103558.583685 mapwrite 0x80c214 thru 0x80d598 (0x1385 bytes) 000005[0] 1765103558.718411 write 0x16b16d thru 0x16beca (0xd5e bytes) 000012[0] 1765103558.839102 write 0x5c42e2 thru 0x5c5232 (0xf51 bytes) 000013[0] 1765103558.886236 mapwrite 0x22a728 thru 0x22b9d5 (0x12ae bytes) truncating to largest ever: 0xab367 000014[1] 1765103558.978570 trunc from 0x80d599 to 0x0ab366 (0x762232 bytes) truncating to largest ever: 0x7ac837 000016[0] 1765103559.066589 trunc from 0x0ab367 to 0x7ac836 (0x7014d0 bytes) 000022[1] 1765103559.197207 mapread 0x6cb62b thru 0x6cc877 (0x124d bytes) 000023[1] 1765103559.242100 trunc from 0x7ac837 to 0x07443b (0x7383fb bytes) 000027[0] 1765103559.396831 mapread 0x0221f7 thru 0x022a33 (0x83d bytes) 000030[0] 1765103559.468378 mapwrite 0x27318d thru 0x27325b (0xcf bytes) 000033[0] 1765103559.689724 trunc from 0x27325c to 0x0d69b9 (0x19c8a2 bytes) 000037[1] 1765103559.709939 mapwrite 0x9efb5d thru 0x9f168a (0x1b2e bytes) 000038[1] 1765103559.815535 read 0x386551 thru 0x387fdf (0x1a8f bytes) 000039[0] 1765103559.846244 mapread 0x23de3f thru 0x23f84c (0x1a0e bytes) 000041[1] 1765103559.937035 write 0x12bd26 thru 0x12c8f8 (0xbd3 bytes) 000042[0] 1765103559.991332 mapread 0x05805e thru 0x0584f7 (0x49a bytes) 000043[1] 1765103560.062234 write 0x1a16b7 thru 0x1a2ddb (0x1725 bytes) 000049[1] 1765103560.172398 mapread 0x57e46f thru 0x57f6cc (0x125e bytes) 000050[0] 1765103560.189440 trunc from 0x9f168b to 0x5d1e64 (0x41f826 bytes) 000052[0] 1765103560.292934 mapwrite 0x7652bd thru 0x765ab9 (0x7fd bytes) 000056[1] 1765103560.444382 mapwrite 0x607509 thru 0x607d77 (0x86f bytes) 000057[0] 1765103560.584133 mapread 0x3d1f01 thru 0x3d22c4 (0x3c4 bytes) 000058[0] 1765103560.614665 write 0x720d2c thru 0x7226f8 (0x19cd bytes) 000061[0] 1765103560.682009 write 0x4ff57e thru 0x500bd7 (0x165a bytes) 000062[1] 1765103560.694014 mapread 0x2210e2 thru 0x2215f4 (0x513 bytes) 000063[0] 1765103560.729943 mapwrite 0x211524 thru 0x211956 (0x433 bytes) 000064[1] 1765103560.804887 write 0x201c0a thru 0x202422 (0x819 bytes) truncating to largest ever: 0x904119 000066[0] 1765103560.907376 trunc from 0x765aba to 0x904118 (0x19e65f bytes) 000068[1] 1765103560.993561 mapread 0x7161b2 thru 0x7178d8 (0x1727 bytes) 000069[1] 1765103561.049458 read 0x71c3d3 thru 0x71de53 (0x1a81 bytes) 000075[1] 1765103561.115614 trunc from 0x904119 to 0x77804b (0x18c0cd bytes) 000076[1] 1765103561.160436 write 0x1b57d6 thru 0x1b6ead (0x16d8 bytes) 000077[1] 1765103561.169972 write 0x79783b thru 0x798783 (0xf49 bytes) 000079[0] 1765103561.203475 mapwrite 0x022f0d thru 0x023b15 (0xc09 bytes) 000082[1] 1765103561.332457 mapread 0x4f249e thru 0x4f41a1 (0x1d04 bytes) 000085[0] 1765103561.406661 mapwrite 0x8917b5 thru 0x8927bb (0x1007 bytes) 000088[0] 1765103561.588708 trunc from 0x8927bc to 0x40770e (0x48b0ad bytes) 000089[0] 1765103561.614763 trunc from 0x40770f to 0x7497fe (0x3420f0 bytes) 000091[0] 1765103561.681661 mapread 0x659bc8 thru 0x65ac91 (0x10ca bytes) 000092[0] 1765103561.709536 mapread 0x5d74d0 thru 0x5d8535 (0x1066 bytes) truncating to largest ever: 0x97108e 000096[0] 1765103561.848681 trunc from 0x7497ff to 0x97108d (0x22788f bytes) 000097[0] 1765103561.906222 trunc from 0x97108e to 0x6eded8 (0x2831b5 bytes) 000103[1] 1765103561.961816 mapwrite 0x6ae2a9 thru 0x6afe57 (0x1baf bytes) 000104[0] 1765103562.045002 write 0x49e5fc thru 0x49f20b (0xc10 bytes) 000106[0] 1765103562.136712 write 0x275405 thru 0x27592a (0x526 bytes) 000109[1] 1765103562.197298 mapwrite 0x904bf3 thru 0x904e12 (0x220 bytes) 000111[1] 1765103562.336262 trunc from 0x904e13 to 0x64b091 (0x2b9d81 bytes) 000114[1] 1765103562.379743 read 0x5f01ac thru 0x5f0ab7 (0x90c bytes) 000115[1] 1765103562.391840 mapwrite 0x1b9aa9 thru 0x1b9d32 (0x28a bytes) 000125[0] 1765103562.563286 mapread 0x1dc49a thru 0x1dce24 (0x98b bytes) 000126[1] 1765103562.596983 mapread 0x21da68 thru 0x21e85a (0xdf3 bytes) 000127[0] 1765103562.647932 mapwrite 0x03335d thru 0x033836 (0x4da bytes) 000128[0] 1765103562.736305 write 0x373788 thru 0x374135 (0x9ae bytes) 000131[0] 1765103562.797131 write 0x53a381 thru 0x53b4a0 (0x1120 bytes) 000135[1] 1765103562.898487 write 0x9b9829 thru 0x9ba94a (0x1122 bytes) 000136[0] 1765103562.961593 read 0x2d258b thru 0x2d2a15 (0x48b bytes) 000137[1] 1765103562.995576 trunc from 0x9ba94b to 0x0645fb (0x95634f bytes) 000138[1] 1765103563.094823 read 0x026c07 thru 0x028a54 (0x1e4e bytes) 000141[0] 1765103563.207720 read 0x0580ff thru 0x059bf5 (0x1af7 bytes) 000143[0] 1765103563.256048 mapread 0x034ca4 thru 0x03594c (0xca9 bytes) 000146[0] 1765103563.338031 read 0x058078 thru 0x059f21 (0x1eaa bytes) 000148[0] 1765103563.377261 write 0x27487f thru 0x275098 (0x81a bytes) 000150[0] 1765103563.431967 mapread 0x02dfcd thru 0x02e7a3 (0x7d7 bytes) 000151[1] 1765103563.497265 mapread 0x25c35f thru 0x25e1a3 (0x1e45 bytes) 000152[0] 1765103563.619960 write 0x383cd4 thru 0x38483b (0xb68 bytes) 000158[1] 1765103563.754413 write 0x549b11 thru 0x54af7b (0x146b bytes) 000159[0] 1765103563.786012 trunc from 0x54af7c to 0x902668 (0x3b76ed bytes) 000162[1] 1765103563.856705 write 0x369790 thru 0x369984 (0x1f5 bytes) 000165[1] 1765103563.974620 read 0x50b267 thru 0x50c6a2 (0x143c bytes) 000170[1] 1765103564.062667 mapread 0x22b2d3 thru 0x22c008 (0xd36 bytes) 000171[1] 1765103564.083895 read 0x6e4610 thru 0x6e49a1 (0x392 bytes) 000173[0] 1765103564.106883 mapwrite 0x581eef thru 0x58228f (0x3a1 bytes) 000174[0] 1765103564.190851 mapread 0x1fd786 thru 0x1fe4e0 (0xd5b bytes) 000175[0] 1765103564.252502 read 0x0c3c89 thru 0x0c4c8f (0x1007 bytes) 000177[0] 1765103564.258502 mapwrite 0x506bde thru 0x508196 (0x15b9 bytes) 000179[0] 1765103564.319477 read 0x824b7c thru 0x824d40 (0x1c5 bytes) 000181[0] 1765103564.324694 write 0x0919b6 thru 0x092c70 (0x12bb bytes) 000182[0] 1765103564.372747 write 0x6aedc1 thru 0x6af8da (0xb1a bytes) 000184[1] 1765103564.405329 read 0x393c79 thru 0x3950e5 (0x146d bytes) 000193[0] 1765103564.487027 read 0x30c10d thru 0x30d4ab (0x139f bytes) 000194[0] 1765103564.535836 read 0x688da3 thru 0x68a1c2 (0x1420 bytes) 000203[0] 1765103564.715359 write 0x7ec579 thru 0x7edd41 (0x17c9 bytes) 000206[0] 1765103564.741845 mapwrite 0x84fac4 thru 0x850d3b (0x1278 bytes) 000208[0] 1765103564.840208 trunc from 0x902669 to 0x153193 (0x7af4d5 bytes) 000209[1] 1765103564.861467 read 0x103922 thru 0x105688 (0x1d67 bytes) 000210[1] 1765103564.883968 write 0x4f3f5a thru 0x4f468d (0x734 bytes) 000216[0] 1765103564.907458 mapread 0x3a1736 thru 0x3a1781 (0x4c bytes) 000217[0] 1765103564.952099 read 0x2012fd thru 0x2032d8 (0x1fdc bytes) 000219[0] 1765103564.979727 mapread 0x0c753c thru 0x0c8056 (0xb1b bytes) 000221[1] 1765103564.997542 mapwrite 0x7a4e7a thru 0x7a5af1 (0xc78 bytes) 000223[1] 1765103565.103043 mapread 0x679599 thru 0x67b4c0 (0x1f28 bytes) 000225[1] 1765103565.170131 write 0x51f52f thru 0x520873 (0x1345 bytes) 000226[1] 1765103565.181455 mapwrite 0x9e1a59 thru 0x9e316c (0x1714 bytes) 000228[0] 1765103565.323917 read 0x393866 thru 0x394e08 (0x15a3 bytes) 000229[1] 1765103565.363372 read 0x6362d4 thru 0x637d61 (0x1a8e bytes) 000231[0] 1765103565.378643 read 0x2b206e thru 0x2b39a1 (0x1934 bytes) 000233[1] 1765103565.405194 read 0x6cac57 thru 0x6caf38 (0x2e2 bytes) 000234[0] 1765103565.426495 mapwrite 0x2ed7f6 thru 0x2ee599 (0xda4 bytes) 000236[1] 1765103565.523677 mapread 0x237ef3 thru 0x238d5a (0xe68 bytes) 000239[1] 1765103565.581257 write 0x7daba6 thru 0x7dbf85 (0x13e0 bytes) 000242[0] 1765103565.626546 trunc from 0x9e316d to 0x791cdf (0x25148d bytes) 000243[0] 1765103565.675992 write 0x3afa04 thru 0x3afe1b (0x418 bytes) 000246[1] 1765103565.714274 read 0x05ce02 thru 0x05d469 (0x668 bytes) 000251[1] 1765103565.810845 write 0x60c440 thru 0x60c62f (0x1f0 bytes) 000253[1] 1765103565.829739 mapread 0x3774b7 thru 0x37920c (0x1d56 bytes) 000255[1] 1765103565.871300 trunc from 0x791ce0 to 0x0509b6 (0x741329 bytes) 000256[1] 1765103565.929312 mapwrite 0x466f0d thru 0x467614 (0x708 bytes) 000260[1] 1765103566.076729 write 0x47b187 thru 0x47b30e (0x188 bytes) 000261[0] 1765103566.079216 read 0x42dfac thru 0x42e65d (0x6b2 bytes) 000263[0] 1765103566.153943 trunc from 0x47b30f to 0x88dbcf (0x4128c1 bytes) 000264[1] 1765103566.239556 read 0x7c4c2e thru 0x7c58ba (0xc8d bytes) 000265[1] 1765103566.279771 write 0x478e90 thru 0x479c96 (0xe07 bytes) 000268[0] 1765103566.362611 mapread 0x0cf94d thru 0x0d05d5 (0xc89 bytes) 000269[1] 1765103566.399605 trunc from 0x88dbd0 to 0x730f59 (0x15cc76 bytes) 000270[1] 1765103566.473024 mapwrite 0x341943 thru 0x343426 (0x1ae4 bytes) 000271[0] 1765103566.544460 mapwrite 0x5539a8 thru 0x5546c2 (0xd1b bytes) 000273[0] 1765103566.715299 read 0x5f828f thru 0x5f9585 (0x12f7 bytes) 000274[1] 1765103566.748526 mapread 0x38aec8 thru 0x38c48f (0x15c8 bytes) 000276[1] 1765103566.819511 read 0x5d8e1e thru 0x5da2d4 (0x14b7 bytes) 000277[0] 1765103566.859865 mapread 0x43bf88 thru 0x43c806 (0x87f bytes) 000280[1] 1765103566.901564 trunc from 0x730f5a to 0x68cfdf (0xa3f7a bytes) 000282[1] 1765103566.938718 read 0x559e8a thru 0x55b773 (0x18ea bytes) 000286[0] 1765103566.998028 trunc from 0x68cfe0 to 0x3d9482 (0x2b3b5d bytes) 000287[0] 1765103567.087582 read 0x168a1a thru 0x1699a5 (0xf8c bytes) 000288[1] 1765103567.207606 trunc from 0x3d9483 to 0x962180 (0x588cfe bytes) 000290[1] 1765103567.366604 write 0x2cc4b0 thru 0x2ccb06 (0x657 bytes) 000292[1] 1765103567.453652 write 0x8dd5b4 thru 0x8de6b6 (0x1103 bytes) 000293[0] 1765103567.518869 read 0x644a95 thru 0x646929 (0x1e95 bytes) 000294[1] 1765103567.546764 read 0x7fc539 thru 0x7fc580 (0x48 bytes) 000295[1] 1765103567.576871 mapwrite 0x0d3d47 thru 0x0d45da (0x894 bytes) 000296[1] 1765103567.653944 write 0x2731aa thru 0x27331f (0x176 bytes) 000297[1] 1765103567.660597 mapwrite 0x67d64e thru 0x67ed9a (0x174d bytes) 000298[0] 1765103567.719993 trunc from 0x962181 to 0x2c0dbc (0x6a13c4 bytes) 000300[0] 1765103567.772684 write 0x1199ad thru 0x11a075 (0x6c9 bytes) 000301[0] 1765103567.787154 read 0x2a92a1 thru 0x2ab1ca (0x1f2a bytes) 000302[0] 1765103567.866968 read 0x0d5e5b thru 0x0d74d5 (0x167b bytes) 000304[0] 1765103567.894296 read 0x289468 thru 0x28979d (0x336 bytes) 000307[1] 1765103567.923744 write 0x680269 thru 0x6806a4 (0x43c bytes) 000309[1] 1765103567.985175 write 0x194325 thru 0x1958af (0x158b bytes) 000310[0] 1765103568.021153 read 0x13c8ce thru 0x13e808 (0x1f3b bytes) 000313[1] 1765103568.134543 mapread 0x2119a8 thru 0x21309c (0x16f5 bytes) 000314[1] 1765103568.205804 trunc from 0x6806a5 to 0x278caa (0x4079fa bytes) 000315[0] 1765103568.271128 write 0x682c59 thru 0x6836ce (0xa76 bytes) 000317[0] 1765103568.325926 mapread 0x1f2850 thru 0x1f3e9e (0x164f bytes) 000318[0] 1765103568.355121 mapread 0x2d7aff thru 0x2d8038 (0x53a bytes) 000322[0] 1765103568.417687 trunc from 0x6836cf to 0x8fee3e (0x27b770 bytes) 000324[1] 1765103568.488631 mapread 0x3cf19e thru 0x3d0b23 (0x1986 bytes) 000326[1] 1765103568.534218 write 0x11c535 thru 0x11d3ef (0xebb bytes) 000327[1] 1765103568.573380 mapread 0x8528b6 thru 0x852b20 (0x26b bytes) 000328[1] 1765103568.623584 mapread 0x069fa3 thru 0x06aede (0xf3c bytes) 000329[1] 1765103568.712963 write 0x1ef65e thru 0x1ef927 (0x2ca bytes) 000332[0] 1765103568.770215 mapwrite 0x267581 thru 0x267a32 (0x4b2 bytes) 000337[0] 1765103568.941189 mapread 0x3cfb52 thru 0x3cfef7 (0x3a6 bytes) 000338[0] 1765103568.950593 trunc from 0x8fee3f to 0x095cb3 (0x86918b bytes) 000343[0] 1765103569.024548 mapread 0x0036df thru 0x0050e9 (0x1a0b bytes) 000344[1] 1765103569.043837 write 0x566f6b thru 0x568276 (0x130c bytes) 000345[1] 1765103569.074303 write 0x1a9669 thru 0x1aa024 (0x9bc bytes) 000348[0] 1765103569.152637 trunc from 0x568277 to 0x0bec49 (0x4a962d bytes) 000351[1] 1765103569.265624 mapread 0x018c18 thru 0x01a64f (0x1a38 bytes) 000352[1] 1765103569.334232 trunc from 0x0bec4a to 0x74a44c (0x68b803 bytes) 000353[1] 1765103569.402542 write 0x5f2525 thru 0x5f3169 (0xc45 bytes) 000354[1] 1765103569.442961 trunc from 0x74a44d to 0x75618a (0xbd3e bytes) 000355[0] 1765103569.492894 trunc from 0x75618b to 0x4bbc6f (0x29a51b bytes) 000356[1] 1765103569.570956 read 0x15354c thru 0x155273 (0x1d28 bytes) 000361[0] 1765103569.651824 trunc from 0x4bbc70 to 0x07add5 (0x440e9a bytes) 000362[1] 1765103569.698219 write 0x46efc9 thru 0x4700cd (0x1105 bytes) 000363[0] 1765103569.726162 read 0x3b7202 thru 0x3b789f (0x69e bytes) 000364[1] 1765103569.756939 mapwrite 0x42e6d2 thru 0x42fb0c (0x143b bytes) 000365[0] 1765103569.828192 trunc from 0x4700ce to 0x3eb818 (0x848b5 bytes) 000367[0] 1765103569.872966 trunc from 0x3eb819 to 0x644f29 (0x259711 bytes) 000368[1] 1765103569.914938 trunc from 0x644f2a to 0x5fbc4e (0x492db bytes) 000372[1] 1765103570.010396 trunc from 0x5fbc4f to 0x2a5808 (0x356446 bytes) 000375[1] 1765103570.060680 mapread 0x1b3c7d thru 0x1b561e (0x19a2 bytes) 000376[1] 1765103570.102924 mapread 0x0aabd5 thru 0x0acaad (0x1ed9 bytes) 000377[1] 1765103570.121744 trunc from 0x2a5809 to 0x399af6 (0xf42ee bytes) 000380[1] 1765103570.163920 trunc from 0x399af7 to 0x7dd285 (0x44378f bytes) 000381[0] 1765103570.190714 read 0x2b821b thru 0x2b94e4 (0x12ca bytes) 000384[1] 1765103570.339678 write 0x03b339 thru 0x03bf10 (0xbd8 bytes) 000386[0] 1765103570.384603 mapread 0x3618c9 thru 0x362884 (0xfbc bytes) 000387[1] 1765103570.449210 mapread 0x326ac9 thru 0x3272b7 (0x7ef bytes) 000393[0] 1765103570.576856 mapwrite 0x98c9f3 thru 0x98d081 (0x68f bytes) 000394[1] 1765103570.811096 trunc from 0x98d082 to 0x63b425 (0x351c5c bytes) 000395[0] 1765103570.942566 write 0x3046b4 thru 0x3061ac (0x1af9 bytes) 000396[0] 1765103571.010873 mapwrite 0x90a2ff thru 0x90b435 (0x1137 bytes) 000397[1] 1765103571.132220 trunc from 0x90b436 to 0x6a29a1 (0x268a94 bytes) 000399[0] 1765103571.176022 trunc from 0x6a29a2 to 0x7d7bad (0x13520c bytes) 000400[0] 1765103571.228569 read 0x29157e thru 0x2924eb (0xf6e bytes) 000401[0] 1765103571.238583 trunc from 0x7d7bae to 0x5adb3b (0x22a072 bytes) 000405[1] 1765103571.291214 write 0x90c71e thru 0x90cf67 (0x84a bytes) 000406[0] 1765103571.336976 write 0x8eccba thru 0x8eea70 (0x1db7 bytes) 000407[1] 1765103571.370549 mapread 0x2d7192 thru 0x2d8f6c (0x1ddb bytes) 000408[0] 1765103571.428905 read 0x44d856 thru 0x44f17c (0x1927 bytes) 000409[1] 1765103571.450838 mapread 0x3e4d58 thru 0x3e603b (0x12e4 bytes) 000410[0] 1765103571.489241 read 0x8af766 thru 0x8afa0f (0x2aa bytes) 000412[1] 1765103571.537621 trunc from 0x90cf68 to 0x83f552 (0xcda15 bytes) 000413[1] 1765103571.633032 read 0x4e5711 thru 0x4e6f44 (0x1834 bytes) 000415[1] 1765103571.666936 mapread 0x6927db thru 0x69453c (0x1d62 bytes) 000417[1] 1765103571.718912 mapread 0x369129 thru 0x369621 (0x4f9 bytes) 000418[1] 1765103571.726385 read 0x678348 thru 0x678538 (0x1f1 bytes) 000420[0] 1765103571.733016 mapwrite 0x3c21c5 thru 0x3c36cf (0x150b bytes) truncating to largest ever: 0x992ecb 000421[1] 1765103571.847520 trunc from 0x83f553 to 0x992eca (0x153978 bytes) 000423[1] 1765103571.924847 read 0x86288f thru 0x864495 (0x1c07 bytes) 000424[0] 1765103571.935730 write 0x22b24f thru 0x22bf28 (0xcda bytes) 000425[1] 1765103571.974730 trunc from 0x992ecb to 0x592329 (0x400ba1 bytes) 000427[1] 1765103572.013657 write 0x36c355 thru 0x36db9b (0x1847 bytes) 000433[1] 1765103572.051207 trunc from 0x59232a to 0x68304e (0xf0d25 bytes) 000435[1] 1765103572.113731 write 0x435328 thru 0x436e65 (0x1b3e bytes) 000436[0] 1765103572.128268 trunc from 0x68304f to 0x8783d9 (0x1f538b bytes) 000437[1] 1765103572.210732 write 0x45edbc thru 0x45f755 (0x99a bytes) 000438[1] 1765103572.279046 mapread 0x2016a6 thru 0x201a2b (0x386 bytes) 000439[1] 1765103572.326495 mapread 0x111051 thru 0x111b48 (0xaf8 bytes) 000440[0] 1765103572.353241 trunc from 0x8783da to 0x2123d8 (0x666001 bytes) 000441[1] 1765103572.429222 mapread 0x17d951 thru 0x17de7e (0x52e bytes) 000443[1] 1765103572.513838 write 0x74a49a thru 0x74a74a (0x2b1 bytes) 000444[1] 1765103572.530682 write 0x71a1fe thru 0x71a57e (0x381 bytes) truncating to largest ever: 0x9aad81 000446[0] 1765103572.582407 trunc from 0x74a74b to 0x9aad80 (0x260636 bytes) 000447[1] 1765103572.664932 mapread 0x45b05e thru 0x45be6a (0xe0d bytes) 000449[1] 1765103572.706317 trunc from 0x9aad81 to 0x4413eb (0x569995 bytes) 000450[1] 1765103572.748566 write 0x061f9e thru 0x0635a0 (0x1603 bytes) 000452[0] 1765103572.760304 trunc from 0x4413ec to 0x631dc5 (0x1f09da bytes) 000453[0] 1765103572.807656 read 0x5c1c20 thru 0x5c248f (0x870 bytes) 000454[0] 1765103572.821884 mapread 0x32be08 thru 0x32dc1e (0x1e17 bytes) 000460[1] 1765103572.963134 mapread 0x3f498c thru 0x3f56f3 (0xd68 bytes) 000463[1] 1765103572.999550 mapread 0x4b1ff4 thru 0x4b3589 (0x1596 bytes) 000465[0] 1765103573.034840 mapread 0x00b240 thru 0x00bfa0 (0xd61 bytes) 000466[0] 1765103573.044794 write 0x75dedb thru 0x75ea22 (0xb48 bytes) 000467[1] 1765103573.060144 mapread 0x61a424 thru 0x61b685 (0x1262 bytes) 000470[0] 1765103573.091836 mapwrite 0x0e9581 thru 0x0ea8ea (0x136a bytes) 000471[1] 1765103573.184825 mapwrite 0x150338 thru 0x151c78 (0x1941 bytes) 000472[1] 1765103573.252892 mapwrite 0x9aa0fc thru 0x9ab2e7 (0x11ec bytes) 000473[0] 1765103573.385237 write 0x8859b5 thru 0x886525 (0xb71 bytes) 000474[0] 1765103573.422944 write 0x3aef4f thru 0x3b09de (0x1a90 bytes) 000475[1] 1765103573.454411 mapwrite 0x82ddcb thru 0x82f621 (0x1857 bytes) 000477[0] 1765103573.557353 mapwrite 0x504270 thru 0x504885 (0x616 bytes) 000478[0] 1765103573.653047 write 0x2b95b5 thru 0x2ba904 (0x1350 bytes) 000485[1] 1765103573.835597 read 0x5ad678 thru 0x5ae1a1 (0xb2a bytes) 000488[1] 1765103573.912912 mapread 0x6e7135 thru 0x6e77af (0x67b bytes) 000490[1] 1765103573.949617 mapread 0x63a809 thru 0x63b925 (0x111d bytes) 000491[1] 1765103573.970239 mapwrite 0x4ba1fb thru 0x4bb63f (0x1445 bytes) 000493[1] 1765103574.066366 mapread 0x48f09f thru 0x490fe7 (0x1f49 bytes) 000497[0] 1765103574.235370 read 0x36103c thru 0x362cf8 (0x1cbd bytes) 000499[1] 1765103574.284649 read 0x31aecc thru 0x31afe5 (0x11a bytes) All operations completed A-OK! PASS 16b (23s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 05:32:59 (1765103579) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 05:33:02 (1765103582) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.362257 s, 23.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.246264 s, 34.1 MB/s ldlm.namespaces.MGC192.168.203.118@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c1bc6e15800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c1bc6e95800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9c1bc6e15800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9c1bc6e95800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e95800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e95800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.66017 s, 12.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.94368 s, 35.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.58753 s, 66.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.66421 s, 22.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.53135 s, 16.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.56638 s, 40.9 MB/s PASS 16d (37s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 05:33:40 (1765103620) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246317 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129342 s, 8.1 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 05:33:46 (1765103626) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 05:34:13 (1765103653) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 05:34:39 (1765103679) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 05:34:44 (1765103684) striped dir -i0 -c2 -H all_char /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 05:34:50 (1765103690) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.81366 s, 14.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.843309 s, 31.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12737 s, 12.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57146 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.19968 s, 11.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.58217 s, 16.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99243 s, 13.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83423 s, 14.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87382 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.62892 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75894 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11173 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.27128 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.73839 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89234 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72191 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.96838 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.1178 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.38989 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (32s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 05:35:23 (1765103723) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun Dec 7 05:36:01 EST 2025 All operations completed A-OK! + fsx 2 done: rc=0 Sun Dec 7 05:36:01 EST 2025 + fsx 3 done: rc=0 Sun Dec 7 05:36:01 EST 2025 All operations completed A-OK! + fsx 4 done: rc=0 Sun Dec 7 05:36:01 EST 2025 + fsx 5 done: rc=0 Sun Dec 7 05:36:01 EST 2025 PASS 16k (46s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 05:36:09 (1765103769) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 05:36:19 (1765103779) mmap test1: basic mmap operation (PASS, 0.097809s) mmap test2: MAP_PRIVATE not write back (PASS, 0.08242s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.897s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4251s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11994s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22708s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.75209s) mmap test11: multi-thread shared mmap (PASS, 0.85752s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1031s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 05:36:48 (1765103808) oleg318-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (3s) == sanityn test 20: test extra readahead page left in cache ============================================================== 05:36:51 (1765103811) striped dir -i0 -c2 -H all_char /mnt/lustre/d20.sanityn PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 05:36:56 (1765103816) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (4s) == sanityn test 23: others should see updated atime while another read============================================================== 05:37:00 (1765103820) now is 1765103822 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 new atime is 1765103883 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 05:38:07 (1765103887) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 483.3K 407 482.9K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 484.6K 333 484.3K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 115.4K 467 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 483.3K 407 482.9K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 484.6K 333 484.3K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 115.4K 467 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.6M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 494902 407 494495 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 496261 333 495928 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 118131 467 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 465 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 483.3K 407 482.9K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 484.6K 333 484.3K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 115.4K 467 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 494902 407 494495 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 496261 333 495928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 118131 467 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117265 465 116800 1% /mnt/lustre[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 494902 407 494495 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 496261 333 495928 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 118131 467 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 465 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (4s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 05:38:12 (1765103892) PASS 24b (4s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 05:38:17 (1765103897) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 05:38:23 (1765103903) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 05:38:29 (1765103909) utime: good mknod times 1765103909 <= 1765103910 <= 1765103910 for /mnt/lustre/f26a utime: good mknod times 1765103909 <= 1765103910 <= 1765103910 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 05:38:35 (1765103915) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 05:38:44 (1765103924) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279604 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-07 05:38:45.000000000 -0500 Birth: 2025-12-07 05:38:46.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279604 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-07 05:38:45.000000000 -0500 Birth: 2025-12-07 05:38:46.000000000 -0500 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 05:38:52 (1765103932) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 05:38:59 (1765103939) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 05:39:05 (1765103945) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.118@tcp:/lustre 7542784 41984 7494656 1% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 05:39:23 (1765103963) -rw-r--r-- 1 root root 0 Dec 7 05:39 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (7s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 05:39:31 (1765103971) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 44951 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 05:39:36 (1765103976) PASS 31t (6s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 05:39:43 (1765103983) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 05:39:46 (1765103986) Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (32s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 05:40:18 (1765104018) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 2 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (37s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 05:40:55 (1765104055) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.27 seconds: 78.67 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.32 seconds: 75.89 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1765104060 ; total 0 ; last 0) total: 100 unlinks in 6 seconds: 16.666666 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1765104066 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (17s) == sanityn test 34: no lock timeout under IO ============= 05:41:13 (1765104073) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid in FULL state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1bc6e15800.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1bc6e15800.ost_server_uuid in IDLE state after 0 sec PASS 34 (108s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 05:43:02 (1765104182) striped dir -i1 -c2 -H all_char /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 54829 54911 for 60 sec... - open/close 876 (time 1765104196.98 total 10.01 last 87.54) - open/close 1787 (time 1765104206.99 total 20.01 last 91.04) - open/close 2664 (time 1765104216.99 total 30.02 last 87.67) - open/close 3548 (time 1765104227.00 total 40.03 last 88.29) total: 4000 open/close in 45.23 seconds: 88.44 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.118@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.118@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (68s) == sanityn test 36: handle ESTALE/open-unlink correctly == 05:44:10 (1765104250) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.787051 s, 66.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.636541 s, 82.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.618045 s, 84.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.629317 s, 83.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.566089 s, 92.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.650766 s, 80.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.586313 s, 89.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.545046 s, 96.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.604827 s, 86.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.693235 s, 75.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.636138 s, 82.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7441408) after(7492608) PASS 36 (239s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 05:48:09 (1765104489) striped dir -i1 -c2 -H crush2 /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 - create 2215 (time 1765104502.23 total 10.01 last 221.38) - create 4440 (time 1765104512.24 total 20.01 last 222.48) - create 7266 (time 1765104522.24 total 30.01 last 282.58) - create 9796 (time 1765104532.24 total 40.01 last 252.99) total: 10000 create in 40.95 seconds: 244.22 ops/second PASS 37 (47s) == sanityn test 39a: file mtime does not change after rename ========================================================== 05:48:56 (1765104536) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 05:48:59 (1765104539) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 05:49:03 (1765104543) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 05:49:07 (1765104547) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 05:49:11 (1765104551) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 05:49:21 (1765104561) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 05:49:29 (1765104569) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 05:49:39 (1765104579) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 05:49:47 (1765104587) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 05:49:53 (1765104593) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 05:49:59 (1765104599) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 05:50:05 (1765104605) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (7s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 05:50:12 (1765104612) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 05:50:17 (1765104617) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 05:50:23 (1765104623) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41g (6s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 05:50:29 (1765104629) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 05:50:35 (1765104635) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (740s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 06:02:55 (1765105375) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 06:03:00 (1765105380) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 06:03:05 (1765105385) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 06:03:11 (1765105391) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 06:03:16 (1765105396) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 06:03:22 (1765105402) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 06:03:27 (1765105407) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 06:03:32 (1765105412) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 06:03:38 (1765105418) PASS 43a (42s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 06:04:20 (1765105460) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=22 Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 06:04:25 (1765105465) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (6s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 06:04:31 (1765105471) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 06:04:36 (1765105476) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 06:04:42 (1765105482) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 06:04:47 (1765105487) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 06:04:53 (1765105493) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 06:04:58 (1765105498) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 43i (6s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 06:05:04 (1765105504) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (39s) == sanityn test 43k: unlink vs create ==================== 06:05:44 (1765105544) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (440s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 06:13:03 (1765105983) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Dec 7 06:13:07 EST 2025 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 06:13:08 (1765105988) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 06:13:14 (1765105994) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 06:13:19 (1765105999) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 06:13:24 (1765106004) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 06:13:29 (1765106009) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 06:13:35 (1765106015) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 06:13:40 (1765106020) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 06:13:45 (1765106025) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (6s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 06:13:51 (1765106031) PASS 45a (48s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 06:14:39 (1765106079) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=45 Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 06:14:45 (1765106085) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 06:14:50 (1765106090) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 06:14:55 (1765106095) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 06:15:00 (1765106100) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 06:15:06 (1765106106) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 06:15:11 (1765106111) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 06:15:16 (1765106116) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (5s) == sanityn test 45j: read vs rename ====================== 06:15:21 (1765106121) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (448s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:22:49 (1765106569) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 06:22:54 (1765106574) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 06:22:59 (1765106579) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:23:05 (1765106585) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 06:23:10 (1765106590) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 06:23:16 (1765106596) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 06:23:21 (1765106601) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 06:23:26 (1765106606) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 06:23:32 (1765106612) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (6s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:23:37 (1765106617) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (6s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:23:43 (1765106623) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:23:49 (1765106629) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:23:55 (1765106635) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:24:00 (1765106640) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (6s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:24:06 (1765106646) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:24:12 (1765106652) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (7s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:24:18 (1765106658) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00580201 s, 1.8 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:24:25 (1765106665) 0+1 records in 0+1 records out 158 bytes copied, 0.00314721 s, 50.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (233115) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:24:29 (1765106669) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0030685 s, 334 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148074 s, 70.8 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:24:44 (1765106684) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00390914 s, 262 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:24:51 (1765106691) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129736 s, 80.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:24:54 (1765106694) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1338 0x53a 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1338 0x53a 0x280000401 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 06:24:58 (1765106698) llite.lustre-ffff9c1bc6e15800.xattr_cache=1 llite.lustre-ffff9c1bc6e95800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762656 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:24:58.000000000 -0500 Modify: 2025-12-07 06:24:58.000000000 -0500 Change: 2025-12-07 06:24:58.000000000 -0500 Birth: 2025-12-07 06:24:59.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281286 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:25:04.000000000 -0500 Modify: 2025-12-07 06:25:04.000000000 -0500 Change: 2025-12-07 06:25:04.000000000 -0500 Birth: 2025-12-07 06:25:05.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762659 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:25:10.000000000 -0500 Modify: 2025-12-07 06:25:10.000000000 -0500 Change: 2025-12-07 06:25:10.000000000 -0500 Birth: 2025-12-07 06:25:10.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281287 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:25:10.000000000 -0500 Modify: 2025-12-07 06:25:10.000000000 -0500 Change: 2025-12-07 06:25:10.000000000 -0500 Birth: 2025-12-07 06:25:10.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281289 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:25:15.000000000 -0500 Modify: 2025-12-07 06:25:16.000000000 -0500 Change: 2025-12-07 06:25:16.000000000 -0500 Birth: 2025-12-07 06:25:16.000000000 -0500 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762660 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:25:15.000000000 -0500 Modify: 2025-12-07 06:25:16.000000000 -0500 Change: 2025-12-07 06:25:16.000000000 -0500 Birth: 2025-12-07 06:25:16.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 06:25:23 (1765106723) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 06:25:30 (1765106730) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 06:25:38 (1765106738) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 06:25:51 (1765106751) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 06:26:01 (1765106761) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (14s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 06:26:14 (1765106774) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 06:26:27 (1765106787) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 06:26:41 (1765106801) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1765106802 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.44156 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (28s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 06:27:09 (1765106829) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1765106831 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.69922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (38s) == sanityn test 60: Verify data_version behaviour ======== 06:27:47 (1765106867) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 06:27:50 (1765106870) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:27:52 (1765106872) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:27:55 (1765106875) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00703002 s, 9.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:27:56 (1765106876) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00501106 s, 13.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:27:58 (1765106878) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:27:59 (1765106879) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00421141 s, 15.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:28:00 (1765106880) llite.lustre-ffff9c1bc6e15800.xattr_cache=1 llite.lustre-ffff9c1bc6e95800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:28:02 (1765106882) llite.lustre-ffff9c1bc6e15800.xattr_cache=1 llite.lustre-ffff9c1bc6e95800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:28:05 (1765106885) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00295189 s, 347 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00307399 s, 333 kB/s 246378: get lock2 246381: get lock1 246381: done 246381: get lock2 246378: done 246378: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 246381: cannot get lock2: Resource deadlock avoided 246381: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 246378: cannot get lock1: Resource deadlock avoided 246378: done 246378: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:28:11 (1765106891) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0247076 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145704 s, 72.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 246970 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 06:28:18 (1765106898) striped dir -i0 -c2 -H all_char /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (48s) == sanityn test 77a: check FIFO NRS policy =============== 06:29:06 (1765106946) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127298 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387301 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544486 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932412 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638924 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973111 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115532 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111993 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101738 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116486 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112404 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109754 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1237 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112032 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133827 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125202 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150679 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015541 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015041 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031212 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128277 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117289 s, 89.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273569 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626721 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398886 s, 263 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389825 s, 26.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431342 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319522 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296563 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233903 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129129 s, 81.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009949 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550117 s, 191 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 06:29:09 (1765106949) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151216 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320194 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537199 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547353 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644758 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693193 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074293 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453474 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961121 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104411 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088181 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105584 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991739 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729263 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799577 s, 13.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103769 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112816 s, 9.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167774 s, 62.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174694 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137772 s, 76.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194459 s, 53.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162309 s, 64.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198548 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246152 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227412 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177919 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239554 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181464 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205768 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200681 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274284 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210959 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235432 s, 44.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133063 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294067 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298691 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998457 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110625 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104958 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106834 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986624 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104327 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104043 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107968 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116304 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117891 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115944 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103583 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115411 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125836 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672853 s, 156 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225214 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228294 s, 45.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190926 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172956 s, 60.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198057 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220142 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179482 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704187 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145679 s, 72.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228423 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388305 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149541 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865018 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676238 s, 155 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682882 s, 154 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 06:29:13 (1765106953) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137331 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393903 s, 26.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296688 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123796 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144818 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162559 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14894 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162916 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137993 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148559 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185088 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149036 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153897 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185298 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15703 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151954 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154441 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175053 s, 59.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300141 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340716 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20748 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305843 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224101 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042896 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540112 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429961 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039429 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502504 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405721 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475963 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477036 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448145 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065867 s, 159 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=3 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191104 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052701 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060016 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679475 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695689 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346015 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399568 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494647 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379969 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931358 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119415 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109362 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120072 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119861 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899083 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859618 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135854 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01506 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138291 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016064 s, 65.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160566 s, 65.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146366 s, 71.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149704 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109837 s, 95.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111963 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171591 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166455 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373406 s, 281 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408799 s, 257 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203795 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574214 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.001906 s, 550 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00170728 s, 614 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131131 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332347 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041169 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717884 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115323 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111694 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119184 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116801 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118857 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11408 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128802 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122644 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122111 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158232 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141216 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158076 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176032 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081815 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265036 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228983 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387257 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216495 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313578 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210138 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316038 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340703 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467042 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290999 s, 36.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373869 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322015 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388742 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040803 s, 25.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399741 s, 26.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 06:29:18 (1765106958) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120728 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256409 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166001 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739325 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806513 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722584 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879733 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732913 s, 14.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834851 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108903 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106805 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117315 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126185 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121056 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107854 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110408 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128013 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163978 s, 63.9 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213091 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134872 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139822 s, 75.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017292 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207137 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165396 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206986 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169229 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029743 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193078 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277781 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126321 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109414 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217128 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824186 s, 127 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171418 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221127 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377071 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251728 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593049 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581877 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066696 s, 15.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048442 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659051 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524904 s, 20.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574026 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00774325 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500607 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25198 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207549 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528237 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605208 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604079 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246628 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00273345 s, 384 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059756 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226067 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635287 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271296 s, 387 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347613 s, 302 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737711 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077727 s, 13.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262241 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800992 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270996 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282635 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283277 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277245 s, 3.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129313 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335798 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491913 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041208 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058138 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561385 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587224 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420587 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696314 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630165 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971779 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854102 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892566 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100492 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105667 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978249 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101448 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162554 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136001 s, 77.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133636 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151231 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100687 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202287 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203896 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155843 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805711 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042809 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191076 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252163 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404991 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621337 s, 169 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310726 s, 337 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286063 s, 36.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 06:29:23 (1765106963) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.18@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.156483 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378813 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371145 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038064 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103121 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103319 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13019 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105093 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10622 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124813 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127193 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145101 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133907 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141808 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140585 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138011 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131207 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22248 s, 4.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231117 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217496 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230686 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229067 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216384 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218194 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236494 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226408 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230921 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237132 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226638 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234369 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225356 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233669 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232094 s, 4.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132104 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436842 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357136 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477302 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587927 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555911 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668207 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548815 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620368 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477691 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055472 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717022 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681731 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800368 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969947 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866096 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104735 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170274 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16685 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166599 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163758 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168249 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165205 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1772 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167108 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175291 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167006 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172083 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168366 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168888 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173563 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171582 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180338 s, 5.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142788 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277775 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554564 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617999 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736175 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598533 s, 17.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760271 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969523 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094859 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108377 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965686 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967089 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100352 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112739 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104781 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133333 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143119 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120282 s, 87.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142212 s, 73.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771098 s, 136 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119714 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162507 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170461 s, 61.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235982 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240752 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198671 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221883 s, 47.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275243 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356327 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311775 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363664 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235791 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241856 s, 434 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147405 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039322 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388736 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756646 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839848 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898058 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123268 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154205 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155156 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160449 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331978 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159145 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158928 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182649 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317816 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186096 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579863 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253397 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177632 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192241 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477156 s, 22.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209157 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274169 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028233 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316783 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217511 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423937 s, 24.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128003 s, 81.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153699 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304784 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175283 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400086 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313069 s, 33.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:29:31 (1765106971) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.203417 s, 82.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459842 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033159 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536876 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802193 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128829 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126907 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123019 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122214 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136048 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143515 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126339 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124908 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155479 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152074 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135651 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151488 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013839 s, 75.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147544 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199763 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477113 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122856 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202839 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211143 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161521 s, 64.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238425 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156368 s, 67.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651499 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134542 s, 77.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00959473 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319204 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125021 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0075924 s, 138 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14202 s, 118 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456919 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713289 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100396 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141561 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147647 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131553 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128237 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148085 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1431 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164464 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148381 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153047 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14602 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157327 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167843 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181951 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215299 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331723 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219432 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025588 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271747 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268606 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309781 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243617 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186477 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147213 s, 71.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215788 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167068 s, 62.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106198 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173571 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258459 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527819 s, 19.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.188889 s, 88.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322128 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418013 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678187 s, 15.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150989 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137857 s, 76.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152043 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184324 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243937 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225104 s, 46.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161133 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032151 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156993 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150196 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144661 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236055 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162105 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152765 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174217 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502705 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160027 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294603 s, 35.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167793 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175636 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189485 s, 55.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178976 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0088904 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138931 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128679 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240645 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157533 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202562 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169434 s, 6.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129995 s, 129 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445926 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362323 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741669 s, 14.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904103 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108377 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123509 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145457 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13459 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00752183 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141878 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149602 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131261 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142905 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294305 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163595 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140757 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174476 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03363 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120741 s, 86.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222663 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229221 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164613 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378596 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463183 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399761 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989653 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606616 s, 17.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6e15800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6e15800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681182 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301698 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620189 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293214 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650023 s, 16.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 06:29:39 (1765106979) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:29:43 (1765106983) oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 06:29:47 (1765106987) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:29:54 (1765106994) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151613 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478409 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893839 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12048 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106858 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145615 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105284 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144424 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153461 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149925 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155619 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145333 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128987 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130218 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149208 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14469 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131305 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186963 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213677 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279999 s, 37.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303653 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261338 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226033 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280336 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204321 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259273 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309291 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107944 s, 97.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178103 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109298 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127321 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117635 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00585872 s, 179 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95432 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4392 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:30:40 (1765107040) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.459479 s, 36.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509531 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584164 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783726 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627589 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862009 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836511 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750933 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567559 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812644 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122973 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12658 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133913 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121262 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142625 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152036 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144018 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411981 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167918 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151564 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172083 s, 60.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187032 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188668 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158708 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181893 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230106 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176773 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195605 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432912 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202449 s, 51.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206361 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236922 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00213253 s, 492 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8323 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.948 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.456531 s, 36.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048722 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838194 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935202 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105237 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118142 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916818 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961561 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116111 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115873 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128989 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115083 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133393 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116015 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124195 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152809 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170947 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210062 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270501 s, 38.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190832 s, 54.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307752 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306788 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024471 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286656 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023611 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395052 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123488 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104644 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191678 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349883 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318187 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296619 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612675 s, 171 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6688 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9472 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (132s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 06:32:53 (1765107173) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453174 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061556 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724555 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756998 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911171 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098239 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132566 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122357 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126523 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119139 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109759 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145576 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12371 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133384 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133321 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151676 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169036 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167268 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188776 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183958 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156273 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209009 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191787 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173913 s, 60.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212154 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160017 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189156 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153079 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036999 s, 283 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222295 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379919 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229484 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022843 s, 459 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8337 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9534 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453622 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570398 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705298 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723901 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769767 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840782 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923887 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717711 s, 14.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764135 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119084 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119573 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137023 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109074 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105659 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118945 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135246 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133272 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183548 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205708 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240601 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192128 s, 54.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230458 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198793 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220537 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254665 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299411 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254868 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284917 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494758 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00867116 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329054 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109279 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00596754 s, 176 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8331 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9583 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:35:06 (1765107306) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255273 s, 65.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763313 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718423 s, 14.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656696 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110476 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125191 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118859 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138611 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152832 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135465 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135321 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135406 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135249 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162744 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154359 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172067 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164129 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183793 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014679 s, 71.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220951 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265885 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235522 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218196 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022257 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284928 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218933 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147372 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956567 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138213 s, 75.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111393 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182674 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672569 s, 156 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9339 s, 21.3 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73499 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.18@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.18@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162486 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432841 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661902 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804824 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748523 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689267 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894153 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10822 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113908 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889994 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883177 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12827 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13804 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169979 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183474 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22748 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242247 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226223 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261841 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306403 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247786 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290188 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352314 s, 29.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170012 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301396 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827101 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304247 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266331 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160674 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343006 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448185 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448371 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113843 s, 92.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9245 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73813 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.18@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142126 s, 118 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022924 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030212 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800437 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844229 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10286 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777463 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880519 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118956 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107516 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112799 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115334 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114212 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110198 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133746 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123952 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130926 s, 8.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152912 s, 68.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229117 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267144 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209908 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02616 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223701 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267494 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839821 s, 125 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249809 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536708 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153334 s, 68.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169911 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606216 s, 173 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137124 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127719 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00505611 s, 207 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9672 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88262 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180714 s, 92.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326256 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326459 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116298 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096467 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103902 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119511 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111201 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102678 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118988 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131945 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127556 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113805 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14213 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132247 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140285 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144345 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241381 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228813 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220975 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246653 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128548 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243549 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253665 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031211 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362204 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138095 s, 75.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144173 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366974 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02355 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246047 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160318 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00802936 s, 131 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77522 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73625 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92029 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73523 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141031 s, 119 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324824 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066912 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793957 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090091 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836788 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100038 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913329 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891978 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976894 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110934 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10932 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129481 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106777 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147711 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144523 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167072 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118174 s, 88.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0085404 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185973 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156025 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011947 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141947 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173286 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239681 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268089 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198267 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166067 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311525 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296687 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551391 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497394 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747976 s, 140 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6748 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4371 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174418 s, 96.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364544 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718568 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786058 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959218 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730851 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790374 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834674 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912763 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126642 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108966 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124389 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116788 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115764 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102465 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887002 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1097 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162263 s, 64.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232612 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170028 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277236 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00869593 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189022 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208621 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219091 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221132 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222684 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277451 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264747 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237985 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688559 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907197 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310732 s, 33.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94908 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7385 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137366 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241969 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571995 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570733 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473693 s, 22.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570302 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709816 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639732 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674648 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065067 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986411 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127443 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993778 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157599 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159636 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224937 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204655 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202637 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225033 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280587 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319518 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273173 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247688 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261572 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257144 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310579 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280753 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265039 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317746 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373269 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281304 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379291 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136119 s, 77.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78571 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73733 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76775 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73204 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (355s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 06:41:01 (1765107661) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254769 s, 65.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017859 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364834 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156292 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167013 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156718 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155188 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181713 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145563 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179718 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188339 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160023 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176335 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172393 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152437 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180394 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173398 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226655 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211235 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225472 s, 46.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202324 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192094 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312616 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250364 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428975 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058759 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338327 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131452 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147561 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229813 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361673 s, 29.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285637 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186929 s, 56.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9544 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73317 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 06:41:37 (1765107697) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.18@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.18@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14255 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199366 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423703 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532756 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492687 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847425 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756152 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901805 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904167 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087927 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906143 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970755 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951586 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174714 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158903 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203629 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218686 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017315 s, 60.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229595 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139384 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186149 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185232 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245249 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025174 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420804 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114416 s, 91.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209398 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301073 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284093 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490857 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790113 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129064 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688247 s, 152 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95278 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7304 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 06:42:18 (1765107738) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.18@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.250592 s, 67.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388369 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332814 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558029 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112681 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961213 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13565 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138877 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128962 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168367 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155874 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170983 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151198 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154426 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148577 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160218 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152053 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182636 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200719 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288066 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231811 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288008 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030413 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198923 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135693 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220775 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00849509 s, 123 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186561 s, 56.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342873 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130155 s, 80.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868927 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344533 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170763 s, 61.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92761 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97162 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 06:42:50 (1765107770) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163918 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471949 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876031 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809839 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101879 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987347 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11992 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121232 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117548 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100005 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10837 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131515 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113782 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135138 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129956 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238763 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229923 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201685 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153924 s, 68.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020297 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172083 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227141 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207402 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023774 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757955 s, 138 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580418 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155472 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151565 s, 69.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204948 s, 51.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319918 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00700807 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262898 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214312 s, 489 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7796 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73486 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91639 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73265 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 06:44:07 (1765107847) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.850713 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237468 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433116 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427243 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678891 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622201 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648798 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582449 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578855 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158395 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178486 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186474 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170854 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209603 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19597 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210077 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187308 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171439 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167655 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198472 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014807 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121404 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172734 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017932 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210689 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205553 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071994 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491842 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550845 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262205 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258309 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266312 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00174353 s, 601 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7455 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9604 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (68s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 06:45:15 (1765107915) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310357 s, 54.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444532 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118584 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135692 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149319 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133275 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131271 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167427 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127795 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148916 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154699 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152725 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162628 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154937 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173054 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147459 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19346 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205639 s, 51.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240395 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218896 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219467 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158601 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212211 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262573 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278472 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388702 s, 270 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198144 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275586 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405331 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040364 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308346 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565436 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920858 s, 114 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95804 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78661 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.562041 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484287 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465491 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709115 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110391 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121718 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128678 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150611 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107317 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15629 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172952 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131889 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179354 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161667 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195639 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20108 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201132 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213502 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258218 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273414 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277644 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340094 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328986 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348769 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423831 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420765 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430435 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374465 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850653 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403219 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074237 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485286 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944767 s, 11.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3074 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87889 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94657 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80289 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (126s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 06:47:21 (1765108041) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre opts: Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg318-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg318-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg318-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg318-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/58170/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg318-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c1bc6751800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1bc6e92000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6751800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6e92000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.302933 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660985 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713973 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169776 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125453 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186068 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178718 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191443 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219382 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186505 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219759 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208109 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219933 s, 4.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20149 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227494 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27173 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242586 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230656 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242539 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316443 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405369 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341421 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357512 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362417 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371933 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393148 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404671 s, 25.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413286 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362647 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437174 s, 24.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630246 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409579 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475448 s, 22.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32209 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00529 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.321314 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812675 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913603 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843743 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944149 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888134 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777304 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882994 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978989 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119002 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147419 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159519 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164592 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168643 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166656 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23053 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242766 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113657 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282523 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208225 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243778 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263837 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275501 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289333 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290324 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278239 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305153 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206099 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247336 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285181 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303928 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288639 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338365 s, 31.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1594 s, 20.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83563 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (170s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 06:50:11 (1765108211) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (10s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:50:21 (1765108221) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (6s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:50:27 (1765108227) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg318-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.74445 s, 280 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg318-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0676 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (53s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:51:20 (1765108280) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224507 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311568 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650381 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756313 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821344 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472571 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920199 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648391 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866815 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141571 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144411 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151134 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137088 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143927 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152891 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183997 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1bc6751800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9c1bc6751800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194793 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015552 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326129 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408894 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070959 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520118 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359225 s, 292 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638547 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479213 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573475 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572084 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441158 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538908 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06355 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560659 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024956 s, 42.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216353 s, 48.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41784 s, 19.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96339 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257762 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566223 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447501 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671768 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382525 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572412 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516214 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813133 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242047 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267839 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277319 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26484 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289254 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269941 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293238 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304974 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332179 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121658 s, 86.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259174 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132936 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201684 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432381 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214033 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425394 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310266 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246863 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040003 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493387 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379975 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318954 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031851 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537876 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558168 s, 18.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.33888 s, 19.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00307 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (72s) == sanityn test 77o: Changing rank should not panic ====== 06:52:32 (1765108352) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (6s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:52:38 (1765108358) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (56s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:53:34 (1765108414) mds.MDS.mdt.nrs_policies=tbf oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg318-client: oleg318-server: ssh exited with exit code 36 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (16s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 06:53:50 (1765108430) striped dir -i0 -c1 -H crush /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 288894 kill md_thread xargs PASS 77r (41s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:54:31 (1765108471) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 06:54:35 (1765108475) striped dir -i1 -c2 -H crush2 /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 12288 Blocks: 24 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639051 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:54:36.000000000 -0500 Modify: 2025-12-07 06:54:36.000000000 -0500 Change: 2025-12-07 06:54:36.000000000 -0500 Birth: 2025-12-07 06:54:36.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:54:48 (1765108488) total: 10 open/close in 0.06 seconds: 170.70 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 Finish migration, then checking.. PASS 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:54:53 (1765108493) start migration thread 296148 accessing the migrating directory for 1 minute... concurrent access 725 failures, 1246 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 296148 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &> /dev/null; done ) (wd: ~) PASS 80b (63s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:55:56 (1765108556) stat: cannot statx 'd0/0': No such file or directory File: d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381367729 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 06:55:57.000000000 -0500 Modify: 2025-12-07 06:55:57.000000000 -0500 Change: 2025-12-07 06:55:57.000000000 -0500 Birth: 2025-12-07 06:55:57.000000000 -0500 PASS 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:55:59 (1765108559) - open/close 1818 (time 1765108570.88 total 10.00 last 181.77) total: 2500 open/close in 14.46 seconds: 172.89 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7638: 299769 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (151s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:58:31 (1765108711) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 06:58:32 (1765108712) - open/close 2164 (time 1765108724.41 total 10.00 last 216.35) total: 2500 open/close in 11.17 seconds: 223.77 ops/second total: 2500 open/close in 9.53 seconds: 262.21 ops/second snapshot_time 1765108779.341279019 secs.nsecs start_time 1765108714.261104492 secs.nsecs elapsed_time 65.080174527 secs.nsecs open 2505 samples [usecs] 124 5761 1490910 1020340190 close 2504 samples [usecs] 70 3294 484740 120519196 mknod 2501 samples [usecs] 215 5654 1300011 792045299 unlink 2500 samples [usecs] 321 31700 2530309 6018344199 mkdir 2 samples [usecs] 929 3484 4413 13001297 rmdir 2 samples [usecs] 3428 4190 7618 29307284 rename 2500 samples [usecs] 265 18905 5558158 20693847098 getattr 3767 samples [usecs] 1 8260 132723 122669605 statfs 13 samples [usecs] 14 45 291 7239 parallel_rename_file 2500 samples [usecs] 265 18905 5558158 20693847098 crossdir_rename 2500 samples [usecs] 265 18905 5558158 20693847098 snapshot_time 1765108779.341381779 secs.nsecs start_time 1765108714.261207544 secs.nsecs elapsed_time 65.080174235 secs.nsecs open 2504 samples [usecs] 124 7980 1428291 983449561 close 2504 samples [usecs] 79 3045 454349 102367101 mknod 2500 samples [usecs] 230 7867 1238912 759882980 unlink 2500 samples [usecs] 310 39983 2475039 5719663681 mkdir 2 samples [usecs] 1458 6717 8175 47243853 rmdir 2 samples [usecs] 1919 8277 10196 72191290 rename 2500 samples [usecs] 281 11916 4362978 11789987946 getattr 3757 samples [usecs] 7 248 108895 3894211 statfs 12 samples [usecs] 11 31 252 5594 parallel_rename_file 2500 samples [usecs] 281 11916 4362978 11789987946 crossdir_rename 2500 samples [usecs] 281 11916 4362978 11789987946 PASS 81d (68s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 06:59:40 (1765108780) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8405 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8405 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 06:59:43 (1765108783) start pid 313984 to create/unlink striped directory start pid 313985 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5942: 313984 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5942: 313985 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (124s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 07:01:47 (1765108907) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 85: Lustre API root cache race =========== 07:01:57 (1765108917) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1765108918 Finishing test test1 at 1765108920 PASS 85 (6s) == sanityn test 90: open/create and unlink striped directory ========================================================== 07:02:03 (1765108923) start pid 319947 to open/create under striped directory start pid 319949 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6009: 319947 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6009: 319949 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (184s) == sanityn test 91: chmod and unlink striped directory === 07:05:07 (1765109107) start pid 324112 to chmod striped directory start pid 324113 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6046: 324112 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6046: 324113 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (185s) == sanityn test 92: create remote directory under orphan directory ========================================================== 07:08:12 (1765109292) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 07:08:16 (1765109296) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 3703 0xe77 0x2c0000400 0 3736 0xe98 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5558 0x15b6 0x280000401 1 5555 0x15b3 0x2c0000401 rc1=0 and rc2=0 PASS 93 (13s) == sanityn test 94: signal vs CP callback race =========== 07:08:29 (1765109309) osc.lustre-OST0000-osc-ffff9c1bc6751800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9c1bc6e92000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c1bc6751800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c1bc6e92000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.133855 s, 61.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00171 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 331110 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.04058 s, 101 kB/s osc.lustre-OST0000-osc-ffff9c1bc6751800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1bc6e92000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6751800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1bc6e92000.idle_timeout=debug PASS 94 (12s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 07:08:41 (1765109321) llite.lustre-ffff9c1bc6751800.fast_read=0 llite.lustre-ffff9c1bc6e92000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0261429 s, 80.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9c1bc6751800.fast_read=1 llite.lustre-ffff9c1bc6e92000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 07:08:56 (1765109336) llite.lustre-ffff9c1bc6751800.fast_read=0 llite.lustre-ffff9c1bc6e92000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00444906 s, 2.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9c1bc6751800.fast_read=1 llite.lustre-ffff9c1bc6e92000.fast_read=1 PASS 95b (10s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 07:09:06 (1765109346) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 07:09:08 (1765109348) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00440657 s, 930 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (4s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 07:09:12 (1765109352) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0891771 s, 23.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 07:09:16 (1765109356) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0944449 s, 22.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (4s) == sanityn test 100e: DoM: read on open and file size ==== 07:09:19 (1765109359) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00498594 s, 2.4 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 07:09:22 (1765109362) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00315452 s, 1.3 MB/s PASS 101a (4s) == sanityn test 101b: Discard DoM data on rename ========= 07:09:26 (1765109366) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00356503 s, 1.1 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 07:09:29 (1765109369) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00337037 s, 1.2 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 07:09:34 (1765109374) striped dir -i1 -c2 -H all_char /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1765109375 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1765109378 Finishing test test23 at 1765109378 PASS 103 (9s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 07:09:43 (1765109383) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 07:09:44 (1765109384) striped dir -i1 -c2 -H all_char /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (19s) == sanityn test 106a: Verify the btime via statx() ======= 07:10:03 (1765109403) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 07:10:05 (1765109405) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00703883 s, 145 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372920 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 07:10:06.000000000 -0500 Modify: 2025-12-07 07:10:06.000000000 -0500 Change: 2025-12-07 07:10:06.000000000 -0500 Birth: 2025-12-07 07:10:06.000000000 -0500 /mnt/lustre/f106b.sanityn 144115272381372920 -rw-r--r-- 1 0 0 1765109406 1765109406 1765109406 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372920 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 07:10:06.000000000 -0500 Modify: 2025-12-07 07:10:06.000000000 -0500 Change: 2025-12-07 07:10:06.000000000 -0500 Birth: 2025-12-07 07:10:06.000000000 -0500 1765109406 1024 1 PASS 106b (4s) == sanityn test 106c: Verify statx attributes mask ======= 07:10:09 (1765109409) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 07:10:12 (1765109412) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.153432 s, 68.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5560 0x15b8 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 First grouplock blocks second one PASS 107a (6s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 07:10:18 (1765109418) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.142671 s, 73.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5564 0x15bc 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8405 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 07:10:29 (1765109429) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00784412 s, 1.3 MB/s llite.lustre-ffff9c1bc6751800.hybrid_io=0 llite.lustre-ffff9c1bc6e92000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.46892 s, 1.9 MB/s PASS 108a (8s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 07:10:37 (1765109437) 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Iteration 9 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 PASS 109 (618s) == sanityn test 110: do not grant another lock on resend ========================================================== 07:20:55 (1765110055) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272431697923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-07 07:20:56.000000000 -0500 Modify: 2025-12-07 07:20:56.000000000 -0500 Change: 2025-12-07 07:20:56.000000000 -0500 Birth: 2025-12-07 07:20:57.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg318-client.virtnet oleg318-client.virtnet PASS 110 (66s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 07:22:01 (1765110121) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6929: 403040 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (6s) == sanityn test 112: update max-inherit in default LMV === 07:22:07 (1765110127) lmv.lustre-clilmv-ffff9c1bc5bf3800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff9c1bc7ce9000.qos_threshold_rr=90 PASS 112 (6s) == sanityn test 113: check servers of specified fs ======= 07:22:13 (1765110133) lustre-OST0000-osc-ffff9c1bc5bf3800 active. lustre-OST0001-osc-ffff9c1bc5bf3800 active. lustre-OST0000-osc-ffff9c1bc5bf3800 active. lustre-OST0001-osc-ffff9c1bc5bf3800 active. MGC192.168.203.118@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 07:22:16 (1765110136) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (9s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 07:22:25 (1765110145) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 07:22:26 (1765110146) llite.lustre-ffff9c1bc5bf3800.intent_mkdir=1 llite.lustre-ffff9c1bc7ce9000.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x1f6:0x0] 1 [0x240000bd2:0x1b7:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (3s) == sanityn test 121: trunc append race =================== 07:22:29 (1765110149) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00374994 s, 5.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162326 s, 1.3 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 07:22:34 (1765110154) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H crush /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0976 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0894 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0994 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.096 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1089 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1068 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1043 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1668 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1862 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1585 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1742 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1832 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.189 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1866 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2699 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2558 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2463 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.286 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1bc5bf3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1bc5bf3800.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 7121 sec ============== 07:23:42 (1765110222) === sanityn: start cleanup 07:23:42 (1765110222) === Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 07:24:52 (1765110292) ===