-----============= acceptance-small: sanity-quota ============----- Fri Feb 14 06:40:44 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg301-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-quota.*ex: No such file or directory excepting tests: 2 4a 63 65 49 skipping tests SLOW=no: 61 12a 9 === sanity-quota: start setup 06:40:53 (1739533253) === oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ae4bd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ae4bd800.idle_timeout=debug oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-quota: finish setup 06:41:03 (1739533263) === using SAVE_PROJECT_SUPPORTED=0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [true] running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [touch] [/mnt/lustre/d0_runas_test/f6988] running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [true] running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [touch] [/mnt/lustre/d0_runas_test/f6988] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 -u quota_2usr space:0 -g quota_usr space:0 -g quota_2usr space:0 -p 1000 space:0 -p 1001 space:0 -u quota_usr inodes:0 -u quota_2usr inodes:0 -g quota_usr inodes:0 -g quota_2usr inodes:0 -p 1000 inodes:0 -p 1001 inodes:0 == sanity-quota test 0: Test basic quota performance ===== 06:41:17 (1739533277) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/f0.sanity-quota-0] [count=10] [conv=fsync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.517941 s, 20.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/f0.sanity-quota-0] [count=10] [conv=fsync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.520508 s, 20.1 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 0 (32s) == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 06:41:49 (1739533309) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:10 MB) -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.279229 s, 18.8 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=5] [seek=5] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0': Disk quota exceeded 5+0 records in 4+0 records out 4702208 bytes (4.7 MB) copied, 0.308592 s, 15.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0689544 s, 0.0 kB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 -------------------------------------- Group quota (block hardlimit:10 MB) -g quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.518135 s, 10.1 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=5] [seek=5] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1': Disk quota exceeded 5+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.340472 s, 12.3 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.100728 s, 0.0 kB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -g quota_usr space:0 -p 1000 space:0 -------------------------------------- Project quota (block hardlimit:10 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.323906 s, 16.2 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=5] [seek=5] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2': Disk quota exceeded 5+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.31955 s, 13.1 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0738412 s, 0.0 kB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -p 1000 space:0 Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1a (118s) == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 06:43:47 (1739533427) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 -u quota_usr space:0 used 0 -u quota_usr bhardlimit:10240 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.496447 s, 10.6 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.407373 s, 12.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0949182 s, 0.0 kB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 -------------------------------------- Group quota (block hardlimit:20 MB) -g quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.291296 s, 18.0 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=5] [seek=5] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1': Disk quota exceeded 5+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.394205 s, 10.6 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.10545 s, 0.0 kB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -g quota_usr space:0 -p 1000 space:0 -------------------------------------- Project quota (block hardlimit:20 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.462616 s, 11.3 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.369999 s, 14.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.101886 s, 0.0 kB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -p 1000 space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1b (132s) == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 06:45:59 (1739533559) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 oleg301-server: Pool lustre.qpool2 created Waiting 90s for '' Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 -u quota_usr space:0 used 0 -u quota_usr bhardlimit:20480 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.811503 s, 12.9 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=10] [seek=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.692574 s, 15.1 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=1] [seek=20] dd: error writing '/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0955351 s, 0.0 kB/s -u quota_usr space:20486 qpool1 used 20486 -u quota_usr space:20486 qpool2 used 20486 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' lustre.qpool2 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg301-server: Pool lustre.qpool2 destroyed Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1c (80s) == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 06:47:19 (1739533639) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 oleg301-server: Pool lustre.qpool2 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' -u quota_usr space:0 used 0 -u quota_usr bhardlimit:20480 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.30022 s, 17.5 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=5] [seek=5] dd: error writing '/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0': Disk quota exceeded 5+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.39379 s, 10.7 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0769439 s, 0.0 kB/s -u quota_usr space:9220 qpool1 used 9220 -u quota_usr space:9220 qpool2 used 9220 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed lustre.qpool2 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg301-server: Pool lustre.qpool2 destroyed Waiting 90s for 'foo' Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1d (72s) == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 06:48:32 (1739533712) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:53000000 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.275131 s, 19.1 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.284843 s, 18.4 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0771143 s, 0.0 kB/s Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-1] [count=20] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.949616 s, 22.1 MB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1e (56s) == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 06:49:28 (1739533768) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:200 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Waiting 90s for '' Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 Waiting 90s for 'lustre-OST0000_UUID ' -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.377841 s, 13.9 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.321948 s, 16.3 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0806363 s, 0.0 kB/s Removing lustre-OST0000_UUID from qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.3296 s, 15.9 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.290702 s, 18.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.080214 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 1f (79s) == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 06:50:47 (1739533847) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=1 User quota (block hardlimit:40 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 -u quota_usr space:0 used 0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.78908 s, 5.9 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=10] [seek=10] dd: error writing '/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0': Disk quota exceeded 9+0 records in 8+0 records out 8523776 bytes (8.5 MB) copied, 1.73117 s, 4.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=2] [seek=20] [oflag=sync] dd: error writing '/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0': Disk quota exceeded 2+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.359389 s, 2.9 MB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -u quota_usr space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=467 Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1g (64s) == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 06:51:51 (1739533911) fallocate on zfs doesn't consume space fallocate not supported SKIP: sanity-quota test_1h need >= 2.13.57 and ldiskfs for fallocate SKIP 1h (1s) == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 06:51:53 (1739533913) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:200 MB) Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Waiting 90s for '' Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.39719 s, 13.2 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.32798 s, 16.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0570887 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 10245 0 0 - 1 0 0 - Pool: lustre.qpool1 lustre-OST0000 10245* - 10245 - 1* - 0 - Total allocated inode limit: 0, total allocated block limit: 10245 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.233854 s, 22.4 MB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.320289 s, 16.4 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.279433 s, 18.8 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0574991 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-1] [count=3] 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.178237 s, 17.6 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2] [count=3] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2': Disk quota exceeded 2+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.124533 s, 8.4 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2] [seek=3] [count=1] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0577318 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 1i (76s) == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 06:53:09 (1739533989) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 -------------------------------------- Project quota (block hardlimit:40 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1 /home/green/git/lustre-release/lustre/utils/lfs project -p 1001 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 40960 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1001 (pid 1001): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 81920 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 1.99574 s, 21.0 MB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=20] [seek=40] [oflag=direct] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 1.03703 s, 20.2 MB/s osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=1 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=1 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=1 osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=1 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=1 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=1 error: get_param: param_path '*/*/quota_slave/root_prj_enable': No such file or directory Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 61449* 0 40960 - 0* 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 61449* - 61449 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 61449 Disk quotas for prj 1001 (pid 1001): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 81920 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1] [count=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.040244 s, 0.0 kB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1] [count=20] [seek=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.032338 s, 0.0 kB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2] [count=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 1.72512 s, 24.3 MB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2] [count=20] [seek=40] [oflag=direct] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.790146 s, 26.5 MB/s osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=0 osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=0 On MGS 192.168.203.101, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.203.101, default.admin_nodemap = nodemap.default.admin_nodemap=1 waiting 10 secs for sync On MGS 192.168.203.101, default.trusted_nodemap = nodemap.default.trusted_nodemap=1 waiting 10 secs for sync On MGS 192.168.203.101, test_1j.admin_nodemap = nodemap.test_1j.admin_nodemap=1 waiting 10 secs for sync On MGS 192.168.203.101, test_1j.trusted_nodemap = nodemap.test_1j.trusted_nodemap=1 waiting 10 secs for sync On MGS 192.168.203.101, test_1j.rbac = nodemap.test_1j.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops waiting 10 secs for sync running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=20] [seek=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0': Disk quota exceeded 2+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0777763 s, 13.5 MB/s On MGS 192.168.203.101, active = nodemap.active=0 waiting 10 secs for sync running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=40] [seek=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 1.73416 s, 24.2 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete -p 1000 space:0 osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=0 osd-zfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-zfs.lustre-OST0001.quota_slave.root_prj_enable=0 Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 1j (125s) SKIP: sanity-quota test_2 skipping excluded test 2 == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 06:55:14 (1739534114) old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=1 User quota (soft limit:4 MB grace:60 seconds) -u quota_usr space:0 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.21117 s, 19.9 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.136226 s, 75.2 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5167* 4096 0 59s 0* 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 6148 - 1 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 6232 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5167 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.163688 s, 62.6 kB/s Grace time is 58s Sleep through grace ... ...sleep 63 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193* 4096 0 expired 2* 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 7172 - 1 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 7256 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0211538 s, 48.4 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00509972 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7217* 4096 0 expired 2* 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172* - 7172 - 1* - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 7256 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7217 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 4096 0 - 1 0 0 - lustre-MDT0000 1* - 1 - 1* - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 84 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.204678 s, 20.5 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Group quota (soft limit:4 MB grace:60 seconds) -g quota_usr space:0 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.194509 s, 21.6 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.125138 s, 81.8 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5129 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129* 4096 0 59s 0* 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5129 - 6148 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6148 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00391069 s, 2.6 MB/s Grace time is 59s Sleep through grace ... ...sleep 64 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193* 4096 0 expired 2* 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 6193 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6193 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00824837 s, 0.0 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00637036 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193* 4096 0 expired 2* 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 6193 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6193 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 44 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 4096 0 - 1 0 0 - lustre-MDT0000 1* - 1 - 1* - 0 - lustre-OST0000 44 - 1068 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 1068 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.165262 s, 25.4 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Project quota (soft limit:4 MB grace:60 sec) -p 1000 space:0 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.212523 s, 19.7 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.114628 s, 89.3 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 6 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 6 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124* 4096 0 59s 0* 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 6148 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6148 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.138227 s, 74.1 kB/s Grace time is 59s Sleep through grace ... ...sleep 64 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6149* 4096 0 expired 1* 0 0 - lustre-MDT0000 2 - 0 - 1 - 0 - lustre-OST0000 6148 - 7172 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 7172 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.017013 s, 60.2 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00499706 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7217 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7217 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7173* 4096 0 expired 1* 0 0 - lustre-MDT0000 2 - 0 - 1 - 0 - lustre-OST0000 7172* - 7172 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 7172 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 4096 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w Block grace time: 1m; Inode grace time: 1w /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2 Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.19843 s, 21.1 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=467 PASS 3a (359s) == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 07:01:13 (1739534473) limit 4 glbl_limit 8 grace 60 glbl_grace 120 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=1 User quota in qpool1(soft limit:4 MB grace:60 seconds) -u quota_usr space:0 Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Waiting 90s for '' Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.162909 s, 25.7 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.217494 s, 47.1 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129 8192 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5129 - 6148 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6148 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5129 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00450873 s, 2.3 MB/s Quota info for qpool1: Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5129* 4096 0 59s 0* 0 0 - Grace time is 59s Sleep through grace ... ...sleep 64 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 8192 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 6193 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6193 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00770346 s, 0.0 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00534275 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 8192 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 6193 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6193 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6193 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6191 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 8192 0 - 1 0 0 - lustre-MDT0000 1* - 1 - 1* - 0 - lustre-OST0000 44 - 1068 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 1068 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 44 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.21803 s, 19.2 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Group quota in qpool1(soft limit:4 MB grace:60 seconds) -g quota_usr space:0 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.182444 s, 23.0 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.13761 s, 74.4 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124 8192 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 6148 - 1 - 0 - lustre-OST0001 0 - 41 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6189 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.146071 s, 70.1 kB/s Quota info for qpool1: Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6148* 4096 0 59s 0* 0 0 - Grace time is 59s Sleep through grace ... ...sleep 64 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6215 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 0 - 1 - 0 - lustre-OST0001 66 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6215 8192 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6148 - 7172 - 1 - 0 - lustre-OST0001 66* - 66 - 1* - 0 - Total allocated inode limit: 0, total allocated block limit: 7238 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0282247 s, 36.3 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00488694 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7239 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172 - 0 - 1 - 0 - lustre-OST0001 66 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7239 8192 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7172* - 7172 - 1* - 0 - lustre-OST0001 66* - 66 - 1* - 0 - Total allocated inode limit: 0, total allocated block limit: 7238 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 66 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 66 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 66 8192 0 - 1 0 0 - lustre-MDT0000 1* - 1 - 1* - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 66* - 66 - 1* - 0 - Total allocated inode limit: 0, total allocated block limit: 66 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.160543 s, 26.1 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Project quota in qpool1(soft:4 MB grace:60 sec) -p 1000 space:0 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.145994 s, 28.7 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.122193 s, 83.8 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 5124 8192 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 5124 - 6148 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 6148 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0976109 s, 105 kB/s Quota info for qpool1: Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6148* 4096 0 59s 0* 0 0 - Grace time is 59s Sleep through grace ... ...sleep 64 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6215 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6213 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6215 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 6213 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 6149 8192 0 - 1 0 0 - lustre-MDT0000 2 - 0 - 1 - 0 - lustre-OST0000 6148 - 7172 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 7172 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0859303 s, 11.9 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0159959 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7239 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7237 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7239 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 7237 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 7173 8192 0 - 1 0 0 - lustre-MDT0000 2 - 0 - 1 - 0 - lustre-OST0000 7172* - 7172 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 7172 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 66 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 66 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 66 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 66 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 8192 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w Block grace time: 2m; Inode grace time: 1w /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2 Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.266395 s, 15.7 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Destroy the created pools: qpool1 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=467 PASS 3b (370s) == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 07:07:23 (1739534843) limit 4 limit2 8 glbl_limit 12 grace1 70 grace2 60 glbl_grace 80 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=1 User quota in qpool2(soft:8 MB grace:60 seconds) -u quota_usr space:0 Creating new pool qpool1 oleg301-server: Pool lustre.qpool1 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Creating new pool qpool2 oleg301-server: Pool lustre.qpool2 created Waiting 90s for '' Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [count=8] 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.495177 s, 16.9 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=8192] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.271259 s, 37.7 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 9263 12288 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 9220 - 10244 - 1 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 10328 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 9263 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 9220 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=9216] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.276121 s, 37.1 kB/s Quota info for qpool2: Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 10287* 8192 0 58s 0* 0 0 - Grace time is 58s Sleep through grace ... ...sleep 63 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 10289 12288 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 10244 - 11268 - 1 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 11352 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 10289 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 10244 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=4096] [seek=10240] dd: error writing '/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0594106 s, 17.2 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=14336] dd: error writing '/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0107342 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 11314 12288 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 11269* - 11269 - 1* - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 11353 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 11314 0 0 - 2 0 0 - lustre-MDT0000 2 - 0 - 2 - 0 - lustre-OST0000 11269 - 0 - 1 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 12288 0 - 1 0 0 - lustre-MDT0000 1* - 1 - 1* - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 84 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 84 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 44 0 0 - 1 0 0 - lustre-MDT0000 1 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 44 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 1m20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [count=8] 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.713233 s, 11.8 MB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg301-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' lustre.qpool2 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg301-server: Pool lustre.qpool2 destroyed osc.lustre-OST0000-osc-ffff8800ae4bd800.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff8800ae4bd800.max_dirty_mb=467 PASS 3c (150s) SKIP: sanity-quota test_4a skipping excluded test 4a == sanity-quota test 4b: Grace time strings handling ===== 07:09:54 (1739534994) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Valid grace strings test Block grace time: 1w3d; Inode grace time: 16m40s Block grace time: 5s; Inode grace time: 1w2d3h4m5s Invalid grace strings test lfs: bad inode-grace: 5c setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT lfs: bad inode-grace: 18446744073709551615 setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT lfs: bad inode-grace: -1 setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT PASS 4b (7s) == sanity-quota test 5: Chown & chgrp successfully even out of block/file quota ========================================================== 07:10:01 (1739535001) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'ugp' Updated after 2s: want 'ugp' got 'ugp' Set quota limit (0 10M 0 10) for quota_usr.quota_usr lfs setquota: warning: inode hardlimit '10' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details lfs setquota: warning: inode hardlimit '10' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details -u quota_usr inodes:0 -g quota_usr inodes:0 -u quota_usr space:0 -g quota_usr space:0 -p 1000 inodes:0 -p 1000 space:0 Create more than 10 files and more than 10 MB ... total: 11 create in 0.06 seconds: 198.13 ops/second /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d5.sanity-quota/f5.sanity-quota-0_1 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.378114 s, 30.5 MB/s Chown files to quota_usr.quota_usr ... - unlinked 0 (time 1739535018 ; total 0 ; last 0) total: 11 unlinks in 0 seconds: inf unlinks/second Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 5 (36s) == sanity-quota test 6: Test dropping acquire request on master ========================================================== 07:10:37 (1739535037) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 -u quota_usr space:0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=1] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108771 s, 9.6 MB/s running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_2usr] [count=1] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0513353 s, 20.4 MB/s at_max=20 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] dd: error writing '/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr': Disk quota exceeded 3+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.34234 s, 6.1 MB/s sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_val=601 fail_loc=0x513 osd-zfs.lustre-OST0000.quota_slave.timeout=10 osd-zfs.lustre-OST0001.quota_slave.timeout=10 running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_2usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.331732 s, 9.5 MB/s Sleep for 41 seconds ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] at_max=600 fail_val=0 fail_loc=0 dd: error writing '/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr': Disk quota exceeded 3+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 54.2464 s, 38.7 kB/s Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 6 (108s) == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 07:12:25 (1739535145) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 -u quota_usr space:0 Waiting 90s for 'none' -u quota_usr --ost 000 bhardlimit:0 Stop ost1... Stopping /mnt/lustre-ost1 (opts:) on oleg301-server Enable quota & set quota limit for quota_usr Waiting 90s for 'ugp' Start ost1... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1095 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1090 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1085 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1080 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1075 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1070 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1065 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1060 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1055 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1050 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1045 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1040 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1035 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1030 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1025 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1020 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1015 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1010 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1005 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1000 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 995 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 990 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 985 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 980 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 975 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 970 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 965 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 960 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 955 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 950 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 945 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 940 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 935 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 930 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 925 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 920 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 915 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 910 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 905 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 900 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 895 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 890 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 885 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 880 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 875 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 870 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 865 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 860 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 855 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 850 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 845 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 840 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 835 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 830 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 825 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 820 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 815 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 810 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 805 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 800 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 795 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 790 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 785 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 780 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 775 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 770 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 765 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 760 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 755 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 750 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 745 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 740 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 735 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 730 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 725 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 720 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 715 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 710 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 705 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 700 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 695 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 690 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 685 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 680 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 675 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 670 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 665 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 660 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 655 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 650 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 645 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 640 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 635 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 630 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 625 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 620 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 615 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 610 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 605 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 600 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 595 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 590 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 585 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 580 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 575 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 570 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 565 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 560 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 555 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 550 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 545 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 540 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 535 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 530 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 525 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 520 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 515 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 510 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 505 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 500 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 495 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 490 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 485 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 480 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 475 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 95 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 90 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 85 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 80 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 75 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 70 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 65 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 60 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 55 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 50 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 45 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 40 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 35 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 30 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 25 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 20 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 15 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 10 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 5 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 0 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status recovery not done in 1475 sec. status: COMPLETE oleg301-server: status: COMPLETE pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1465 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1460 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1455 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1450 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1445 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1440 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1435 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1430 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1425 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1420 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1415 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1410 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1405 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1400 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1395 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1390 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1385 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1380 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1375 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1370 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1365 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1360 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1355 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1350 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1345 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1340 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1335 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1330 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1325 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1320 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1315 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1310 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1305 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1300 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1295 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1290 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1285 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1280 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1275 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1270 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1265 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1260 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1255 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1250 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1245 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1240 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1235 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1230 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1225 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1220 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1215 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1210 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1205 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1200 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1195 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1190 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1185 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1180 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1175 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1170 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1165 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1160 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1155 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1150 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1145 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1140 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1135 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1130 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1125 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1120 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1115 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1110 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1105 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1100 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1095 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1090 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1085 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1080 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1075 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1070 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1065 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1060 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1055 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1050 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1045 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1040 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1035 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1030 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1025 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1020 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1015 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1010 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1005 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 1000 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 995 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 990 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 985 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 980 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 975 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 970 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 965 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 960 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 955 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 950 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 945 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 940 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 935 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 930 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 925 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 920 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 915 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 910 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 905 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 900 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 895 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 890 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 885 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 880 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 875 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 870 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 865 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 860 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 855 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 850 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 845 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 840 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 835 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 830 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 825 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 820 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 815 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 810 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 805 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 800 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 795 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 790 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 785 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 780 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 775 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 770 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 765 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 760 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 755 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 750 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 745 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 740 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 735 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 730 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 725 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 720 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 715 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 710 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 705 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 700 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 695 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 690 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 685 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 680 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 675 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 670 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 665 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 660 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 655 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 650 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 645 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 640 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 635 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 630 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 625 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 620 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 615 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 610 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 605 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 600 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 595 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 590 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 585 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 580 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 575 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 570 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 565 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 560 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 555 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 550 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 545 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 540 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 535 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 530 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 525 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 520 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 515 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 510 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 505 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 500 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 495 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 490 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 485 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 480 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 475 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 470 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 465 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 460 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 455 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 450 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 445 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 440 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 435 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 430 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 425 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 420 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 415 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 410 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 405 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 400 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 395 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 390 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 385 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 380 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 375 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 370 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 365 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 360 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 355 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 350 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 345 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 340 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 335 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 330 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 325 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 320 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 315 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 310 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 305 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 300 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 295 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 290 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 285 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 280 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 275 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 270 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 265 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 260 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 255 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 250 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 245 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 240 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 235 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 230 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 225 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 220 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 215 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 210 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 205 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 200 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 195 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 190 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 185 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 180 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 175 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 170 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 165 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 160 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 155 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 150 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 145 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 140 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 135 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 130 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 125 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 120 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 115 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 110 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 105 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 100 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 95 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 90 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 85 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 80 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 75 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 70 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 65 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 60 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 55 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 50 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 45 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 40 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 35 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 30 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 25 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 20 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 15 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 10 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 5 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg301-server: Waiting 0 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg301-server: status: INACTIVE oleg301-server: *.lustre-OST0001.recovery_status recovery not done in 1475 sec. status: INACTIVE oleg301-server: status: INACTIVE pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1095 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1090 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1085 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1080 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1075 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1070 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1065 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1060 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1055 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1050 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1045 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1040 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1035 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1030 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1025 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1020 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1015 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1010 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1005 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 1000 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 995 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 990 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 985 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 980 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 975 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 970 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 965 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 960 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 955 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 950 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 945 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 940 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 935 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 930 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 925 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 920 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 915 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 910 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 905 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 900 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 895 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 890 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 885 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 880 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 875 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg301-server: Waiting 870 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg301-server: status: COMPLETE oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE