== sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 20:25:14 (1718929514) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'u' User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 sysctl: cannot stat /proc/sys/lustre/timeout: No such file or directory Write 100M (buffered) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2972 1284716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1904 1285784 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3597276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7202908 1% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.1743 s, 48.2 MB/s Fail mds for 0 seconds Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server reboot facets: mds1 Failover mds1 to oleg448-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=12437, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000_UUID 0 - 0 - 1 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 102400 - 114688 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 114688 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 sysctl: cannot stat /proc/sys/lustre/timeout: No such file or directory Write 100M (directio) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] [oflag=direct] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1904 1285784 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3596136 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7201768 1% /mnt/lustre Fail mds for 0 seconds Failing mds1 on oleg448-server 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 3.13582 s, 33.4 MB/s Stopping /mnt/lustre-mds1 (opts:) on oleg448-server reboot facets: mds1 Failover mds1 to oleg448-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=14866, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000_UUID 0 - 0 - 1 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 102400 - 109568 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 109568 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete