-----============= acceptance-small: sanity-quota ============----- Fri Feb 14 06:44:28 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg116-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-quota.*ex: No such file or directory excepting tests: 2 4a 63 65 skipping tests SLOW=no: 61 oleg116-server: debugfs 1.46.2.wc5 (26-Mar-2022) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 === sanity-quota: start setup 06:44:34 (1739533474) === oleg116-client.virtnet: executing check_config_client /mnt/lustre oleg116-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg116-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a702000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a702000.idle_timeout=debug oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-quota: finish setup 06:44:42 (1739533482) === using SAVE_PROJECT_SUPPORTED=0 oleg116-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg116-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg116-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg116-server: debugfs 1.46.2.wc5 (26-Mar-2022) running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [true] running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [touch] [/mnt/lustre/d0_runas_test/f7614] running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [true] running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [touch] [/mnt/lustre/d0_runas_test/f7614] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 -u quota_2usr space:0 -g quota_usr space:0 -g quota_2usr space:0 -p 1000 space:0 -p 1001 space:0 -u quota_usr inodes:0 -u quota_2usr inodes:0 -g quota_usr inodes:0 -g quota_2usr inodes:0 -p 1000 inodes:0 -p 1001 inodes:0 == sanity-quota test 0: Test basic quota performance ===== 06:44:53 (1739533493) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/f0.sanity-quota-0] [count=10] [conv=fsync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.298513 s, 35.1 MB/s Waiting 90s for 'ugp' Updated after 2s: want 'ugp' got 'ugp' running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/f0.sanity-quota-0] [count=10] [conv=fsync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.294996 s, 35.5 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 0 (16s) == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 06:45:09 (1739533509) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:10 MB) -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.12946 s, 40.5 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.148476 s, 35.3 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0411309 s, 0.0 kB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 -------------------------------------- Group quota (block hardlimit:10 MB) -g quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.134056 s, 39.1 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.12828 s, 40.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0509452 s, 0.0 kB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -g quota_usr space:0 -p 1000 space:0 -------------------------------------- Project quota (block hardlimit:10 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.132945 s, 39.4 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.121435 s, 43.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0535678 s, 0.0 kB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -p 1000 space:0 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1a (70s) == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 06:46:20 (1739533580) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 -u quota_usr space:0 used 0 -u quota_usr bhardlimit:10240 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.143178 s, 36.6 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.143593 s, 36.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0475668 s, 0.0 kB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 -------------------------------------- Group quota (block hardlimit:20 MB) -g quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.137533 s, 38.1 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.143451 s, 36.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.069612 s, 0.0 kB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -g quota_usr space:0 -p 1000 space:0 -------------------------------------- Project quota (block hardlimit:20 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.141538 s, 37.0 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.145691 s, 36.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1b.sanity-quota/f1b.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0485472 s, 0.0 kB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -p 1000 space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1b (82s) == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 06:47:41 (1739533661) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Creating new pool qpool2 oleg116-server: Pool lustre.qpool2 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 -u quota_usr space:0 used 0 -u quota_usr bhardlimit:20480 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.264572 s, 39.6 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=10] [seek=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.272635 s, 38.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0] [count=1] [seek=20] dd: error writing '/mnt/lustre/d1c.sanity-quota/f1c.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0413205 s, 0.0 kB/s -u quota_usr space:20484 qpool1 used 20484 -u quota_usr space:20484 qpool2 used 20484 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed lustre.qpool2 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg116-server: Pool lustre.qpool2 destroyed Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1c (48s) == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 06:48:29 (1739533709) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:20 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 oleg116-server: Pool lustre.qpool2 created Waiting 90s for '' Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 -u quota_usr space:0 used 0 -u quota_usr bhardlimit:20480 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.128613 s, 40.8 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.14188 s, 37.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1d.sanity-quota/f1d.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0441924 s, 0.0 kB/s -u quota_usr space:10240 qpool1 used 10240 -u quota_usr space:10240 qpool2 used 10240 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed lustre.qpool2 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg116-server: Pool lustre.qpool2 destroyed Waiting 90s for 'foo' Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1d (50s) == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 06:49:19 (1739533759) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:53000000 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Waiting 90s for 'lustre-OST0001_UUID ' -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.147961 s, 35.4 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.130939 s, 40.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0413031 s, 0.0 kB/s Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1e.sanity-quota/f1e.sanity-quota-1] [count=20] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.519018 s, 40.4 MB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1e (39s) == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 06:49:58 (1739533798) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:200 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.13191 s, 39.7 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.134484 s, 39.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0505782 s, 0.0 kB/s Removing lustre-OST0000_UUID from qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.134013 s, 39.1 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.125123 s, 41.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1f.sanity-quota//f1f.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0859932 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1f (52s) == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 06:50:50 (1739533850) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=1 User quota (block hardlimit:40 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 -u quota_usr space:0 used 0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.41869 s, 7.4 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=10] [seek=10] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.6003 s, 6.6 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0] [count=2] [seek=20] [oflag=sync] dd: error writing '/mnt/lustre/d1g.sanity-quota/f1g.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0473628 s, 0.0 kB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=467 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1g (41s) == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 06:51:31 (1739533891) keep default fallocate mode: 0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:10 MB) -u quota_usr space:0 Write 5MiB Using Fallocate running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [fallocate] [-l5MiB] [/mnt/lustre/d1h.sanity-quota/f1h.sanity-quota-0] Write 11MiB Using Fallocate running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [fallocate] [-l11MiB] [/mnt/lustre/d1h.sanity-quota/f1h.sanity-quota-0] fallocate: fallocate failed: Disk quota exceeded Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -u quota_usr space:0 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1h (21s) == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 06:51:52 (1739533912) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:200 MB) Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Waiting 90s for '' Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 -u quota_usr space:0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.128104 s, 40.9 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.119816 s, 43.8 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0383218 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 10240 0 0 - 1 0 0 - Pool: lustre.qpool1 lustre-OST0000 10240* - 10240 - 1* - 0 - Total allocated inode limit: 0, total allocated block limit: 10240 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-1] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.143627 s, 36.5 MB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.120723 s, 43.4 MB/s Write out of block quota ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=5] [seek=5] 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.116879 s, 44.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0] [count=1] [seek=10] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0394895 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-1] [count=3] 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.100385 s, 31.3 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2] [count=3] 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0768326 s, 40.9 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2] [seek=3] [count=1] dd: error writing '/mnt/lustre/d1i.sanity-quota//f1i.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0412746 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1i (52s) == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 06:52:44 (1739533964) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 -------------------------------------- Project quota (block hardlimit:40 mb) /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1 /home/green/git/lustre-release/lustre/utils/lfs project -p 1001 /mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 40960 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1001 (pid 1001): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 81920 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.759471 s, 55.2 MB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=20] [seek=40] [oflag=direct] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.449893 s, 46.6 MB/s osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=1 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=1 error: get_param: param_path '*/*/quota_slave/root_prj_enable': No such file or directory Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 61440* 0 40960 - 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 61440* - 61440 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 61440 Disk quotas for prj 1001 (pid 1001): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 81920 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 61440 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 61440 running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1] [count=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0177712 s, 0.0 kB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1] [count=20] [seek=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0217512 s, 0.0 kB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2] [count=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.592695 s, 70.8 MB/s running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-2] [count=20] [seek=40] [oflag=direct] 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.293564 s, 71.4 MB/s osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=0 On MGS 192.168.201.116, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.116, default.admin_nodemap = nodemap.default.admin_nodemap=1 waiting 10 secs for sync On MGS 192.168.201.116, default.trusted_nodemap = nodemap.default.trusted_nodemap=1 waiting 10 secs for sync On MGS 192.168.201.116, test_1j.admin_nodemap = nodemap.test_1j.admin_nodemap=1 waiting 10 secs for sync On MGS 192.168.201.116, test_1j.trusted_nodemap = nodemap.test_1j.trusted_nodemap=1 waiting 10 secs for sync On MGS 192.168.201.116, test_1j.rbac = nodemap.test_1j.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops waiting 10 secs for sync running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=20] [seek=40] [oflag=direct] dd: error writing '/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0214 s, 0.0 kB/s On MGS 192.168.201.116, active = nodemap.active=0 waiting 10 secs for sync running as uid/gid/euid/egid 0/0/0/0, groups: 0 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d1j.sanity-quota/f1j.sanity-quota-0] [count=40] [seek=40] [oflag=direct] 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.712003 s, 58.9 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 -p 1000 space:0 osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-MDT0001.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0000.quota_slave.root_prj_enable=0 osd-ldiskfs.lustre-OST0001.quota_slave.root_prj_enable=0 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 1j (100s) SKIP: sanity-quota test_2 skipping excluded test 2 == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 06:54:25 (1739534065) old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=1 User quota (soft limit:4 MB grace:20 seconds) -u quota_usr space:0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.156354 s, 26.8 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0361051 s, 284 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148* 4096 0 20s 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 4120 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 4168 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0327483 s, 313 kB/s Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160* 4096 0 expired 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 4132 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 4180 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0': Disk quota exceeded 2+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00835312 s, 123 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00393076 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4164* 4096 0 expired 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4124 - 4132 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 4180 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4164 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4124 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 4096 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 48 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.110899 s, 37.8 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Group quota (soft limit:4 MB grace:20 seconds) -g quota_usr space:0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.101797 s, 41.2 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0106993 s, 957 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148* 4096 0 21s 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 4160 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4160 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0034754 s, 2.9 MB/s Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160* 4096 0 expired 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 4168 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4168 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=4096] [seek=6144] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00464798 s, 0.0 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00389874 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160* 4096 0 expired 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 4168 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4168 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 4096 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 1064 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 1064 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.157706 s, 26.6 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Project quota (soft limit:4 MB grace:20 sec) -p 1000 space:0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.151427 s, 27.7 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0101902 s, 1.0 MB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4108* 4096 0 21s 1* 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 4160 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4160 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00581681 s, 1.8 MB/s Grace time is 21s Sleep through grace ... ...sleep 26 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4120* 4096 0 expired 1* 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 4160 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4160 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=4096] [seek=6144] 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.578332 s, 7.3 MB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00408369 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8216* 4096 0 expired 1* 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216* - 8216 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 8216 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 4096 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w Block grace time: 20s; Inode grace time: 1w /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2 Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3a.sanity-quota/f3a.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.154163 s, 27.2 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=467 PASS 3a (166s) == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 06:57:11 (1739534231) limit 4 glbl_limit 8 grace 20 glbl_grace 40 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=1 User quota in qpool1(soft limit:4 MB grace:20 seconds) -u quota_usr space:0 Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.10818 s, 38.8 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0163644 s, 626 kB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 8192 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 4176 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4176 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00651458 s, 1.6 MB/s Quota info for qpool1: Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160* 4096 0 20s 2* 0 0 - Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 8192 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 4176 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4176 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=4096] [seek=6144] 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.635196 s, 6.6 MB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00676276 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256* 8192 0 40s 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8256* - 8256 - 2* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 8256 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8256 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 8192 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 1064 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 1064 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.118679 s, 35.3 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Group quota in qpool1(soft limit:4 MB grace:20 seconds) -g quota_usr space:0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.114534 s, 36.6 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.009052 s, 1.1 MB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 8192 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 4176 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 4224 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00395431 s, 2.6 MB/s Quota info for qpool1: Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160* 4096 0 20s 2* 0 0 - Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 8192 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 4176 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 4224 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=4096] [seek=6144] 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.607453 s, 6.9 MB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00347253 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256* 8192 0 40s 2* 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216* - 8216 - 1* - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 8264 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 8192 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 48 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-1] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.114889 s, 36.5 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Project quota in qpool1(soft:4 MB grace:20 sec) -p 1000 space:0 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.124001 s, 33.8 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0100894 s, 1.0 MB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4148 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4148 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4108 8192 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4108 - 4176 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4176 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=5120] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00397845 s, 2.6 MB/s Quota info for qpool1: Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4120* 4096 0 20s 1* 0 0 - Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4160 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4160 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 4120 8192 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 4120 - 4176 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 4176 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=4096] [seek=6144] 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.638798 s, 6.6 MB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [bs=1K] [count=10] [seek=10240] dd: error writing '/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00472295 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8256 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8256 - 0 - 2 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8216* 8192 0 40s 1* 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216* - 8216 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 8216 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 40 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 8192 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w Block grace time: 40s; Inode grace time: 1w /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2 Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3b.sanity-quota/f3b.sanity-quota-2] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.114317 s, 36.7 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Destroy the created pools: qpool1 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=467 PASS 3b (176s) == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 07:00:07 (1739534407) limit 4 limit2 8 glbl_limit 12 grace1 30 grace2 20 glbl_grace 40 old_mdmb 467 mdmb 1 osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=1 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=1 User quota in qpool2(soft:8 MB grace:20 seconds) -u quota_usr space:0 Creating new pool qpool1 oleg116-server: Pool lustre.qpool1 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 oleg116-server: Pool lustre.qpool2 created Waiting 90s for '' Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.qpool2 Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [count=8] 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.210943 s, 39.8 MB/s Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=8192] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00678566 s, 1.5 MB/s mmap write when over soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [multiop] [/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0.mmap] [OT40960SMW] Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8244 12288 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8204 - 8224 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 8272 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8244 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8204 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=9216] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00341416 s, 3.0 MB/s Quota info for qpool2: Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256* 8192 0 20s 2* 0 0 - Grace time is 20s Sleep through grace ... ...sleep 25 seconds Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 12288 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 8224 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 8272 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write after timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=4096] [seek=10240] dd: error writing '/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00493033 s, 0.0 kB/s Write after cancel lru locks running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [bs=1K] [count=10] [seek=14336] dd: error writing '/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00367285 s, 0.0 kB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 12288 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 8224 - 1 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 8272 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 8256 0 0 - 2 0 0 - lustre-MDT0000 0 - 0 - 2 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 8216 - 0 - 1 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Unlink file to stop timer Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 12288 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 48 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 48 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 40 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 40 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Block grace time: 40s; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Block grace time: 1w; Inode grace time: 1w Write ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3c.sanity-quota/f3c.sanity-quota-0] [count=8] 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.215735 s, 38.9 MB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Destroy the created pools: qpool1,qpool2 lustre.qpool1 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool1 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool1 oleg116-server: Pool lustre.qpool1 destroyed lustre.qpool2 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.qpool2 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.qpool2 oleg116-server: Pool lustre.qpool2 destroyed osc.lustre-OST0000-osc-ffff88012a702000.max_dirty_mb=467 osc.lustre-OST0001-osc-ffff88012a702000.max_dirty_mb=467 PASS 3c (73s) SKIP: sanity-quota test_4a skipping excluded test 4a == sanity-quota test 4b: Grace time strings handling ===== 07:01:20 (1739534480) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Valid grace strings test Block grace time: 1w3d; Inode grace time: 16m40s Block grace time: 5s; Inode grace time: 1w2d3h4m5s Invalid grace strings test lfs: bad inode-grace: 5c setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT lfs: bad inode-grace: 18446744073709551615 setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT lfs: bad inode-grace: -1 setquota failed: Unknown error -4 Set filesystem quotas. usage: setquota [-t] {-u|-U|-g|-G|-p|-P ID} {-b|-B|-i|-I LIMIT} [--pool POOL] MOUNT_POINT setquota {-u|-g|-p ID} {--default|--delete} MOUNT_POINT PASS 4b (3s) == sanity-quota test 5: Chown & chgrp successfully even out of block/file quota ========================================================== 07:01:23 (1739534483) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'ugp' Set quota limit (0 10M 0 10) for quota_usr.quota_usr lfs setquota: warning: inode hardlimit '10' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details lfs setquota: warning: inode hardlimit '10' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details -u quota_usr inodes:0 -g quota_usr inodes:0 -u quota_usr space:0 -g quota_usr space:0 -p 1000 inodes:0 -p 1000 space:0 Create more than 10 files and more than 10 MB ... total: 11 create in 0.03 seconds: 409.65 ops/second /home/green/git/lustre-release/lustre/utils/lfs project -p 1000 /mnt/lustre/d5.sanity-quota/f5.sanity-quota-0_1 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.256653 s, 44.9 MB/s Chown files to quota_usr.quota_usr ... - unlinked 0 (time 1739534493 ; total 0 ; last 0) total: 11 unlinks in 0 seconds: inf unlinks/second Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 5 (22s) == sanity-quota test 6: Test dropping acquire request on master ========================================================== 07:01:45 (1739534505) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 -u quota_usr space:0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=1] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0371726 s, 28.2 MB/s running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_2usr] [count=1] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244361 s, 42.9 MB/s at_max=20 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] dd: error writing '/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr': Disk quota exceeded 3+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.132892 s, 15.8 MB/s Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 fail_val=601 fail_loc=0x513 osd-ldiskfs.lustre-OST0000.quota_slave.timeout=10 osd-ldiskfs.lustre-OST0001.quota_slave.timeout=10 running as uid/gid/euid/egid 60001/60001/60001/60001, groups: 60001 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_2usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.129355 s, 24.3 MB/s Sleep for 41 seconds ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr] [count=3] [seek=1] [oflag=sync] [conv=notrunc] at_max=600 fail_val=0 fail_loc=0 dd: error writing '/mnt/lustre/d6.sanity-quota/f6.sanity-quota-quota_usr': Disk quota exceeded 3+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 54.8384 s, 38.2 kB/s Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 PASS 6 (84s) == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 07:03:09 (1739534589) Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Creating test directory fail_val=0 fail_loc=0 -u quota_usr space:0 Waiting 90s for 'none' -u quota_usr --ost 000 bhardlimit:0 Stop ost1... Stopping /mnt/lustre-ost1 (opts:) on oleg116-server Enable quota & set quota limit for quota_usr Waiting 90s for 'ugp' Start ost1... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1095 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1090 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1085 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1080 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1075 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1070 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1065 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1060 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1055 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1050 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1045 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1040 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1035 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1030 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1025 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1020 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1015 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1010 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1005 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1000 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 995 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 990 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 985 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 980 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 975 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 970 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 965 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 960 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 955 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 950 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 945 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 940 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 935 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 930 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 925 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 920 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 915 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 910 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 905 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 900 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 895 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 890 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 885 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 880 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 875 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 870 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 865 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 860 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 855 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 850 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 845 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 840 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 835 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 830 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 825 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 820 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 815 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 810 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 805 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 800 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 795 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 790 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 785 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 780 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 775 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 770 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 765 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 760 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 755 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 750 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 745 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 740 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 735 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 730 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 725 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 720 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 715 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 710 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 705 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 700 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 695 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 690 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 685 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 680 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 675 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 670 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 665 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 660 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 655 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 650 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 645 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 640 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 635 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 630 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 625 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 620 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 615 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 610 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 605 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 600 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 595 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 590 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 585 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 580 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 575 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 570 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 565 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 560 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 555 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 550 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 545 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 540 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 535 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 530 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 525 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 520 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 515 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 510 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 505 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 500 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 495 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 490 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 485 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 480 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 475 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 95 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 90 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 85 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 80 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 75 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 70 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 65 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 60 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 55 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 50 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 45 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 40 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 35 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 30 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 25 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 20 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 15 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 10 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 5 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 0 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status recovery not done in 1475 sec. status: COMPLETE oleg116-server: status: COMPLETE pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 affected facets: ost2 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1465 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1460 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1455 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1450 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1445 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1440 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1435 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1430 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1425 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1420 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1415 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1410 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1405 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1400 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1395 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1390 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1385 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1380 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1375 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1370 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1365 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1360 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1355 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1350 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1345 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1340 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1335 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1330 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1325 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1320 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1315 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1310 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1305 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1300 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1295 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1290 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1285 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1280 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1275 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1270 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1265 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1260 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1255 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1250 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1245 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1240 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1235 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1230 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1225 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1220 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1215 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1210 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1205 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1200 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1195 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1190 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1185 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1180 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1175 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1170 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1165 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1160 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1155 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1150 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1145 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1140 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1135 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1130 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1125 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1120 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1115 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1110 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1105 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1100 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1095 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1090 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1085 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1080 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1075 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1070 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1065 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1060 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1055 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1050 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1045 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1040 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1035 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1030 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1025 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1020 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1015 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1010 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1005 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 1000 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 995 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 990 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 985 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 980 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 975 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 970 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 965 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 960 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 955 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 950 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 945 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 940 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 935 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 930 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 925 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 920 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 915 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 910 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 905 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 900 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 895 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 890 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 885 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 880 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 875 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 870 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 865 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 860 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 855 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 850 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 845 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 840 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 835 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 830 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 825 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 820 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 815 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 810 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 805 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 800 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 795 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 790 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 785 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 780 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 775 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 770 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 765 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 760 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 755 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 750 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 745 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 740 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 735 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 730 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 725 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 720 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 715 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 710 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 705 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 700 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 695 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 690 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 685 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 680 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 675 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 670 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 665 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 660 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 655 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 650 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 645 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 640 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 635 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 630 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 625 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 620 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 615 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 610 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 605 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 600 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 595 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 590 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 585 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 580 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 575 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 570 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 565 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 560 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 555 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 550 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 545 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 540 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 535 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 530 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 525 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 520 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 515 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 510 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 505 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 500 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 495 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 490 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 485 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 480 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 475 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 470 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 465 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 460 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 455 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 450 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 445 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 440 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 435 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 430 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 425 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 420 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 415 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 410 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 405 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 400 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 395 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 390 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 385 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 380 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 375 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 370 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 365 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 360 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 355 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 350 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 345 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 340 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 335 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 330 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 325 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 320 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 315 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 310 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 305 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 300 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 295 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 290 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 285 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 280 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 275 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 270 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 265 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 260 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 255 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 250 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 245 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 240 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 235 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 230 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 225 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 220 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 215 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 210 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 205 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 200 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 195 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 190 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 185 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 180 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 175 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 170 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 165 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 160 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 155 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 150 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 145 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 140 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 135 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 130 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 125 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 120 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 115 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 110 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 105 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 100 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 95 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 90 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 85 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 80 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 75 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 70 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 65 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 60 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 55 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 50 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 45 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 40 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 35 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 30 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 25 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 20 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 15 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 10 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 5 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status status: INACTIVE status: INACTIVE oleg116-server: Waiting 0 secs for *.lustre-OST0001.recovery_status recovery done. status: INACTIVE oleg116-server: status: INACTIVE oleg116-server: *.lustre-OST0001.recovery_status recovery not done in 1475 sec. status: INACTIVE oleg116-server: status: INACTIVE pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1380 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1375 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1370 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1365 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1360 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1355 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1350 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1345 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1340 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1335 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1330 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1325 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1320 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1315 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1310 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1305 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1300 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1295 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1290 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1285 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1280 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1275 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1270 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1265 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1260 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1255 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1250 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1245 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1240 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1235 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1230 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1225 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1220 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1215 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1210 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1205 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1200 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1195 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1190 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1185 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1180 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1175 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1170 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1165 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1160 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1155 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1150 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1145 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1140 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1135 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1130 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1125 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1120 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1115 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1110 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1105 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1100 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1095 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1090 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1085 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1080 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1075 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1070 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1065 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1060 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1055 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1050 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1045 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1040 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1035 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1030 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1025 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1020 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1015 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1010 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1005 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 1000 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 995 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 990 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 985 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 980 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 975 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 970 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 965 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 960 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 955 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 950 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 945 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 940 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 935 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 930 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 925 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 920 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 915 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 910 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 905 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 900 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 895 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 890 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 885 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 880 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE oleg116-server: Waiting 875 secs for *.lustre-OST0000.recovery_status recovery done. status: COMPLETE oleg116-server: status: COMPLETE oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE status: COMPLETE