== sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 07:26:51 (1781609211) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'u' User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 Write 100M (buffered) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1472 3601428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2880 7207040 1% /mnt/lustre Fail mds for 40 seconds Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 07:27:03 (1781609223) shut down facet: mds1 facet_host: oleg614-server facet_failover_host: oleg614-server Failover mds1 to oleg614-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 18.7707 s, 5.6 MB/s oleg614-server: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 07:27:19 (1781609239) targets are mounted 07:27:19 (1781609239) facet_failover done oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=124468, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-OST0000 102400* - 81920 - 1* - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 81920 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr space:0 Write 100M (directio) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] [oflag=direct] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1472 3585636 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2880 7191248 1% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.48262 s, 70.7 MB/s Fail mds for 40 seconds Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 07:27:41 (1781609261) shut down facet: mds1 facet_host: oleg614-server facet_failover_host: oleg614-server Failover mds1 to oleg614-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 07:27:56 (1781609276) targets are mounted 07:27:56 (1781609276) facet_failover done oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=126863, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-OST0000 102400 - 109568 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 109568 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete