== sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 10:58:20 (1788533900) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 Write 100M (buffered) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1268640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1628 3568312 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1624 3605396 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3252 7173708 1% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.91524 s, 26.8 MB/s Fail mds for 40 seconds Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 10:59:07 (1788533947) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 10:59:39 (1788533979) targets are mounted 10:59:39 (1788533979) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=130069, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 102400 - 114688 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 114688 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 Write 100M (directio) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] [oflag=direct] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1268724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22108 3583864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1624 3605396 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23732 7189260 1% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.58188 s, 22.9 MB/s Fail mds for 40 seconds Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 11:00:49 (1788534049) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 11:01:18 (1788534078) targets are mounted 11:01:18 (1788534078) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=132469, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 102400 - 109568 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 109568 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete