== sanity-pfl test 22d: out of/low on space + failed to repeat + forced extension ========================================================== 11:21:16 (1754407276) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d22d.sanity-pfl Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. Creating new pool test_22d oleg228-server: Pool lustre.test_22d created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_22d DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 67108864] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=67108864 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x40, extent=[67108864, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 67108864] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 67108864], curr=[67108864, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x40 DEBUG: Checking start <= end: start=67108864, end=18446744073709551615 DEBUG: Component validation passed, continuing UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3716 1283972 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3488 1284200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 34960 3572060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35924 3571096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 70884 7143156 1% /mnt/lustre 348+0 records in 348+0 records out 364904448 bytes (365 MB, 348 MiB) copied, 3.37487 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118434 s, 88.5 MB/s Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. set OST0 lwm back to 3, hwm back to 7 Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 11:22:30 (1754407350) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 11:22:47 (1754407367) targets are mounted 11:22:47 (1754407367) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 67108864] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=67108864 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x40, extent=[67108864, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 67108864] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 67108864], curr=[67108864, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x40 DEBUG: Checking start <= end: start=67108864, end=18446744073709551615 DEBUG: Component validation passed, continuing UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3676 1284012 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3444 1284244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 35984 3571036 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35924 3571096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 71908 7142132 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130102 s, 80.6 MB/s Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. set OST0 lwm back to 3, hwm back to 7 Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 11:23:25 (1754407405) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 11:23:41 (1754407421) targets are mounted 11:23:41 (1754407421) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: test_22d lustre.test_22d oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_22d oleg228-server: Pool lustre.test_22d destroyed