-----============= acceptance-small: replay-single ============----- Wed Jan 15 18:14:43 EST 2025 excepting tests: 110f 131b 59 oleg117-client.virtnet: executing check_config_client /mnt/lustre oleg117-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg117-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b62f9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b62f9000.idle_timeout=debug disable quota as required oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 osd-ldiskfs.track_declares_assert=1 == replay-single test 0a: empty replay =================== 18:14:59 (1736982899) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (35s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 18:15:34 (1736982934) Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.15 seconds: 134.73 ops/second - unlinked 0 (time 1736982957 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (25s) == replay-single test 0c: check replay-barrier =========== 18:15:59 (1736982959) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (104s) == replay-single test 0d: expired recovery with no clients ========================================================== 18:17:43 (1736983063) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre PASS 0d (100s) == replay-single test 1: simple create =================== 18:19:23 (1736983163) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (33s) == replay-single test 2a: touch ========================== 18:19:56 (1736983196) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (33s) == replay-single test 2b: touch ========================== 18:20:29 (1736983229) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (33s) == replay-single test 2c: setstripe replay =============== 18:21:02 (1736983262) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (34s) == replay-single test 2d: setdirstripe replay ============ 18:21:36 (1736983296) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (32s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 18:22:08 (1736983328) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (32s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 18:22:40 (1736983360) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (29s) == replay-single test 3b: replay failed open -ENOMEM ===== 18:23:09 (1736983389) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (33s) == replay-single test 3c: replay failed open -ENOMEM ===== 18:23:42 (1736983422) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (29s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 18:24:11 (1736983451) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (33s) == replay-single test 4b: |x| rm 10 files ================ 18:24:44 (1736983484) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7210928 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (33s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 18:25:17 (1736983517) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (45s) == replay-single test 6a: mkdir + contained create ======= 18:26:03 (1736983563) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (33s) == replay-single test 6b: |X| rmdir ====================== 18:26:36 (1736983596) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (31s) == replay-single test 7: mkdir |X| contained create ====== 18:27:07 (1736983627) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (28s) == replay-single test 8: creat open |X| close ============ 18:27:35 (1736983655) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (29s) == replay-single test 9: |X| create (same inum/gen) ====== 18:28:04 (1736983684) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (28s) == replay-single test 10: create |X| rename unlink ======= 18:28:32 (1736983712) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (29s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 18:29:01 (1736983741) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1424 3605596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre new old Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (28s) == replay-single test 12: open, unlink |X| close ========= 18:29:29 (1736983769) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (28s) == replay-single test 13: open chmod 0 |x| write close === 18:29:57 (1736983797) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (29s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 18:30:26 (1736983826) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (27s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 18:30:53 (1736983853) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (29s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 18:31:22 (1736983882) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (28s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 18:31:50 (1736983910) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (28s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 18:32:19 (1736983939) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 pid: 24678 will close Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (29s) == replay-single test 19: mcreate, open, write, rename === 18:32:48 (1736983968) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1428 3605592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre old Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (27s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:33:15 (1736983995) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1432 3605588 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2844 7211196 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (29s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 18:33:44 (1736984024) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1122 0x462 0 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 1.50978 s, 27.1 MB/s Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg117-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 2844, after 2844 PASS 20b (31s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 18:34:15 (1736984055) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 -rw-r--r-- 1 root root 1 Jan 15 18:34 /mnt/lustre/f20c.replay-single oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 PASS 20c (5s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:34:20 (1736984060) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1432 3605536 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2844 7211144 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (25s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:34:45 (1736984085) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1432 3605560 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2844 7211168 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (29s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:35:14 (1736984114) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (28s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 18:35:42 (1736984142) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (28s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:36:10 (1736984170) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (28s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:36:38 (1736984198) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (28s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:37:06 (1736984226) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (28s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:37:34 (1736984254) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (29s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:38:03 (1736984283) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (28s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:38:31 (1736984311) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (28s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 18:38:59 (1736984339) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (29s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 18:39:28 (1736984368) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 PASS 32 (5s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 18:39:33 (1736984373) total: 10 open/close in 0.03 seconds: 294.05 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.03 seconds: 386.06 ops/second PASS 33a (16s) == replay-single test 33b: test fid seq allocation ======= 18:39:49 (1736984389) fail_loc=0x1311 total: 10 open/close in 0.03 seconds: 368.22 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.03 seconds: 363.25 ops/second PASS 33b (17s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 18:40:06 (1736984406) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 first stat failed: 5 PASS 34 (18s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 18:40:24 (1736984424) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (18s) == replay-single test 36: don't resend cancel ============ 18:40:42 (1736984442) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 36 (27s) == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 18:41:09 (1736984469) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (20s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:41:29 (1736984489) total: 800 open/close in 2.91 seconds: 274.60 ops/second - unlinked 0 (time 1736984494 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1984 1285704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1736984524 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (41s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:42:10 (1736984530) total: 800 open/close in 2.84 seconds: 281.69 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1436 3605584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211192 1% /mnt/lustre - unlinked 0 (time 1736984538 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1736984564 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (39s) == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 18:42:49 (1736984569) SKIP: replay-single test_40 layout_lock needs MDS connection for IO SKIP 40 (1s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:42:50 (1736984570) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00180997 s, 2.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0116826 s, 351 kB/s PASS 41 (3s) == replay-single test 42: recovery after ost failure ===== 18:42:53 (1736984573) total: 800 open/close in 2.53 seconds: 315.77 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2012 1285676 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1736984580 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second debug=-1 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 wait for MDS to timeout and recover debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck - unlinked 0 (time 1736984638 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (67s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:44:00 (1736984640) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (38s) == replay-single test 44a: race in target handle connect ========================================================== 18:44:38 (1736984678) at_max=40 1 of 10 (1736984679) service : cur 1 worst 1 (at 1736982842, 1837s ago) 1 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 2 of 10 (1736984684) service : cur 5 worst 5 (at 1736984684, 0s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 3 of 10 (1736984690) service : cur 5 worst 5 (at 1736984684, 6s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 4 of 10 (1736984695) service : cur 5 worst 5 (at 1736984684, 11s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 5 of 10 (1736984701) service : cur 5 worst 5 (at 1736984684, 17s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 6 of 10 (1736984706) service : cur 5 worst 5 (at 1736984684, 22s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 7 of 10 (1736984712) service : cur 5 worst 5 (at 1736984684, 28s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 8 of 10 (1736984717) service : cur 5 worst 5 (at 1736984684, 33s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 9 of 10 (1736984723) service : cur 5 worst 5 (at 1736984684, 38s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 10 of 10 (1736984728) service : cur 5 worst 5 (at 1736984684, 44s ago) 5 1 1 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (58s) == replay-single test 44b: race in target handle connect ========================================================== 18:45:36 (1736984736) 1 of 10 (1736984736) service : cur 5 worst 5 (at 1736984684, 52s ago) 5 1 1 1 fail_loc=0x80000704 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.117@tcp:/lustre 7666232 2852 7211188 1% /mnt/lustre 2 of 10 (1736984778) service : cur 40 worst 40 (at 1736984776, 2s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 3 of 10 (1736984785) service : cur 40 worst 40 (at 1736984776, 9s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 4 of 10 (1736984786) service : cur 40 worst 40 (at 1736984776, 10s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 5 of 10 (1736984787) service : cur 40 worst 40 (at 1736984776, 11s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 6 of 10 (1736984788) service : cur 40 worst 40 (at 1736984776, 12s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 7 of 10 (1736984789) service : cur 40 worst 40 (at 1736984776, 13s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 8 of 10 (1736984790) service : cur 40 worst 40 (at 1736984776, 14s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 9 of 10 (1736984791) service : cur 40 worst 40 (at 1736984776, 15s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress 10 of 10 (1736984792) service : cur 40 worst 40 (at 1736984776, 16s ago) 40 1 1 1 fail_loc=0x80000704 df: '/mnt/lustre': Operation already in progress PASS 44b (60s) == replay-single test 44c: race in target handle connect ========================================================== 18:46:36 (1736984796) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre total: 100 create in 0.32 seconds: 313.33 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 first stat failed: 5 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (48s) == replay-single test 45: Handle failed close ============ 18:47:24 (1736984844) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 /mnt/lustre/f45.replay-single has type file OK PASS 45 (3s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:47:28 (1736984848) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (49s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:48:17 (1736984897) total: 20 open/close in 0.17 seconds: 117.34 ops/second Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.19 seconds: 106.42 ops/second - unlinked 0 (time 1736984987 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (92s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:49:49 (1736984989) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre total: 20 open/close in 0.26 seconds: 78.27 ops/second Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000216 total: 20 open/close in 0.13 seconds: 159.14 ops/second - unlinked 0 (time 1736985078 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (91s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:51:20 (1736985080) PASS 50 (8s) == replay-single test 52: time out lock replay (3764) ==== 18:51:28 (1736985088) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7490 fail_loc=0x80000157 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (82s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:52:50 (1736985170) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (41s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:53:31 (1736985211) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7490 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (33s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:54:04 (1736985244) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (32s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:54:36 (1736985276) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (29s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:55:05 (1736985305) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (33s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:55:38 (1736985338) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (32s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:56:10 (1736985370) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (32s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 18:56:42 (1736985402) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (34s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 18:57:16 (1736985436) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (59s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 18:58:16 (1736985496) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (40s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 18:58:56 (1736985536) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg117-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (33s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 18:59:29 (1736985569) fail_loc=0x8000012c - open/close 2174 (time 1736985581.12 total 10.00 last 217.34) total: 2500 open/close in 11.46 seconds: 218.10 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1736985626 ; total 0 ; last 0) total: 2500 unlinks in 6 seconds: 416.666656 unlinks/second PASS 58a (64s) == replay-single test 58b: test replay of setxattr op ==== 19:00:33 (1736985633) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) oleg117-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (40s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 19:01:13 (1736985673) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 58c (121s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 19:03:15 (1736985795) total: 200 open/close in 0.51 seconds: 392.75 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1736985798 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1736985822 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (29s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 19:03:44 (1736985824) total: 800 open/close in 1.42 seconds: 563.06 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1440 3605580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1736985829 ; total 0 ; last 0) total: 800 unlinks in 0 seconds: inf unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (80s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 19:05:05 (1736985905) fail_loc=0x8000013a Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00231957 s, 1.8 MB/s PASS 61b (53s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 19:05:58 (1736985958) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (32s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 19:06:30 (1736985990) Stopping /mnt/lustre-mds1 (opts:) on oleg117-server fail_loc=0x80000605 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (9s) == replay-single test 62: don't mis-drop resent replay === 19:06:39 (1736985999) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2856 7211184 1% /mnt/lustre total: 25 open/close in 0.07 seconds: 377.02 ops/second fail_loc=0x80000707 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1736986079 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (81s) == replay-single test 65a: AT: verify early replies ====== 19:08:00 (1736986080) at_history=8 at_history=8 debug=other fail_val=6000 fail_loc=0x8000050a 00000100:00001000:3.0:1736986104.536260:0:13260:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@ffff8800a7b1d0c0 x1821358462262464/t0(0) o101->lustre-MDT0000-mdc-ffff8800a8d73000@192.168.201.117@tcp:12/10 lens 664/66264 e 1 to 0 dl 1736986135 ref 2 fl Rpc:PQr/0/ffffffff rc 0/-1 job:'createmany.0' portal 12 : cur 31 worst 40 (at 1736984778, 1331s ago) 40 40 40 40 portal 29 : cur 1 worst 1 (at 1736983159, 2950s ago) 1 1 0 0 portal 23 : cur 1 worst 1 (at 1736983159, 2950s ago) 1 1 1 1 portal 30 : cur 1 worst 1 (at 1736983164, 2945s ago) 1 0 0 1 portal 17 : cur 1 worst 1 (at 1736983265, 2844s ago) 1 1 1 1 portal 12 : cur 6 worst 40 (at 1736984778, 1341s ago) 40 40 40 40 portal 29 : cur 1 worst 1 (at 1736983159, 2960s ago) 1 1 0 0 portal 23 : cur 1 worst 1 (at 1736983159, 2960s ago) 1 1 1 1 portal 30 : cur 1 worst 1 (at 1736983164, 2955s ago) 1 0 0 1 portal 17 : cur 1 worst 1 (at 1736983265, 2854s ago) 1 1 1 1 PASS 65a (40s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 19:08:40 (1736986120) at_history=8 at_history=8 debug=other trace fail_val=6 fail_loc=0x224 fail_loc=0 00000100:00001000:1.0:1736986144.162537:0:1887:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@ffff8800aa5e4c00 x1821358462271168/t0(0) o4->lustre-OST0000-osc-ffff8800a8d73000@192.168.201.117@tcp:6/4 lens 4584/448 e 1 to 0 dl 1736986175 ref 2 fl Rpc:Qr/0/ffffffff rc 0/-1 job:'multiop.0' debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck portal 28 : cur 1 worst 1 (at 1736983159, 2990s ago) 1 1 1 1 portal 7 : cur 1 worst 1 (at 1736983161, 2988s ago) 1 1 1 1 portal 17 : cur 1 worst 1 (at 1736983230, 2919s ago) 1 0 1 1 portal 6 : cur 31 worst 31 (at 1736986149, 0s ago) 31 1 0 0 PASS 65b (30s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 19:09:10 (1736986150) at_history=8 at_history=8 portal 12 : cur 1 worst 40 (at 1736984778, 1394s ago) 1 40 40 40 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 31 worst 40 (at 1736984778, 1400s ago) 31 40 40 40 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 10 worst 40 (at 1736984778, 1410s ago) 31 40 40 40 fail_loc=0 portal 12 : cur 1 worst 40 (at 1736984778, 1420s ago) 31 40 40 40 Current MDT timeout 1, worst 40 PASS 66a (49s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 19:09:59 (1736986199) at_history=8 at_history=8 fail_val=6 fail_loc=0x50c fail_loc=0 network timeout orig 1, cur 6, worst 6 PASS 66b (47s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 19:10:46 (1736986246) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (53s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 19:11:39 (1736986299) at_history=8 at_history=8 Creating to objid 5697 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.04 seconds: 489.18 ops/second Connected clients: oleg117-client.virtnet oleg117-client.virtnet service : cur 1 worst 1 (at 1736982852, 3470s ago) 1 0 1 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg117-client.virtnet oleg117-client.virtnet service : cur 1 worst 1 (at 1736982852, 3471s ago) 1 0 1 1 0 osc reconnect attempts on 2nd slow PASS 67b (26s) == replay-single test 68: AT: verify slowing locks ======= 19:12:05 (1736986325) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (68s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 19:13:13 (1736986393) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (1s) == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:13:14 (1736986394) Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg117-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19465 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1285400 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3592360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25992 3556308 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7148668 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3593160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26488 3568736 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27944 7161896 1% /mnt/lustre oleg117-client.virtnet: looking for dbench program oleg117-client.virtnet: /usr/bin/dbench oleg117-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg117-client.virtnet oleg117-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg117-client.virtnet' oleg117-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg117-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg117-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg117-client.virtnet at Wed Jan 15 19:13:15 EST 2025 oleg117-client.virtnet: waiting for dbench pid 19490 oleg117-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg117-client.virtnet: oleg117-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg117-client.virtnet: failed to create barrier semaphore oleg117-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg117-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg117-client.virtnet: releasing clients oleg117-client.virtnet: 1 556 19.10 MB/sec warmup 1 sec latency 10.439 ms oleg117-client.virtnet: 1 1236 11.54 MB/sec warmup 2 sec latency 10.308 ms oleg117-client.virtnet: 1 1980 8.82 MB/sec warmup 3 sec latency 9.608 ms oleg117-client.virtnet: 1 2602 7.25 MB/sec warmup 4 sec latency 199.706 ms oleg117-client.virtnet: 1 3768 7.95 MB/sec warmup 5 sec latency 8.728 ms oleg117-client.virtnet: 1 4117 6.68 MB/sec warmup 6 sec latency 248.319 ms oleg117-client.virtnet: 1 4117 5.73 MB/sec warmup 7 sec latency 1248.533 ms oleg117-client.virtnet: 1 4117 5.01 MB/sec warmup 8 sec latency 2248.734 ms oleg117-client.virtnet: 1 4117 4.45 MB/sec warmup 9 sec latency 3248.927 ms oleg117-client.virtnet: 1 4117 4.01 MB/sec warmup 10 sec latency 4249.153 ms oleg117-client.virtnet: 1 4117 3.64 MB/sec warmup 11 sec latency 5249.408 ms oleg117-client.virtnet: 1 4117 3.34 MB/sec warmup 12 sec latency 6249.604 ms oleg117-client.virtnet: 1 4117 3.08 MB/sec warmup 13 sec latency 7249.787 ms oleg117-client.virtnet: 1 4117 2.86 MB/sec warmup 14 sec latency 8249.961 ms oleg117-client.virtnet: 1 4117 2.67 MB/sec warmup 15 sec latency 9250.151 ms oleg117-client.virtnet: 1 4117 2.51 MB/sec warmup 16 sec latency 10250.375 ms oleg117-client.virtnet: 1 4117 2.36 MB/sec warmup 17 sec latency 11250.515 ms oleg117-client.virtnet: 1 4117 2.23 MB/sec warmup 18 sec latency 12250.695 ms oleg117-client.virtnet: 1 4117 2.11 MB/sec warmup 19 sec latency 13250.872 ms oleg117-client.virtnet: 1 4117 2.00 MB/sec warmup 20 sec latency 14251.071 ms oleg117-client.virtnet: 1 4117 1.91 MB/sec warmup 21 sec latency 15251.237 ms oleg117-client.virtnet: 1 4117 1.82 MB/sec warmup 22 sec latency 16251.442 ms oleg117-client.virtnet: 1 4117 1.74 MB/sec warmup 23 sec latency 17251.605 ms oleg117-client.virtnet: 1 4117 1.67 MB/sec warmup 24 sec latency 18251.743 ms oleg117-client.virtnet: 1 4117 1.60 MB/sec warmup 25 sec latency 19251.896 ms oleg117-client.virtnet: 1 4117 1.54 MB/sec warmup 26 sec latency 20252.077 ms oleg117-client.virtnet: 1 4117 1.48 MB/sec warmup 27 sec latency 21252.259 ms oleg117-client.virtnet: 1 4145 1.43 MB/sec warmup 28 sec latency 22186.467 ms oleg117-client.virtnet: 1 4714 1.45 MB/sec warmup 29 sec latency 10.566 ms oleg117-client.virtnet: 1 5368 1.51 MB/sec warmup 30 sec latency 11.562 ms oleg117-client.virtnet: 1 6278 1.65 MB/sec warmup 31 sec latency 7.524 ms oleg117-client.virtnet: 1 7341 1.84 MB/sec warmup 32 sec latency 13.964 ms oleg117-client.virtnet: 1 7883 1.81 MB/sec warmup 33 sec latest_70b fail mds2 2 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36420 3568960 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49364 7162044 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 tency 10.515 ms oleg117-client.virtnet: 1 8590 1.89 MB/sec warmup 34 sec latency 10.426 ms oleg117-client.virtnet: 1 9236 1.85 MB/sec warmup 35 sec latency 9.220 ms oleg117-client.virtnet: 1 10016 1.99 MB/sec warmup 36 sec latency 9.145 ms oleg117-client.virtnet: 1 10876 2.11 MB/sec warmup 37 sec latency 11.096 ms oleg117-client.virtnet: 1 11397 2.07 MB/sec warmup 38 sec latency 13.962 ms oleg117-client.virtnet: 1 11885 2.06 MB/sec warmup 39 sec latency 12.402 ms oleg117-client.virtnet: 1 12450 2.09 MB/sec warmup 40 sec latency 14.299 ms oleg117-client.virtnet: 1 13067 2.08 MB/sec warmup 41 sec latency 10.355 ms oleg117-client.virtnet: 1 14048 2.21 MB/sec warmup 42 sec latency 10.410 ms oleg117-client.virtnet: 1 14606 2.27 MB/sec warmup 43 sec latency 13.383 ms oleg117-client.virtnet: 1 15188 2.24 MB/sec warmup 44 sec latency 12.620 ms oleg117-client.virtnet: 1 15970 2.28 MB/sec warmup 45 sec latency 8.529 ms oleg117-client.virtnet: 1 16693 2.29 MB/sec warmup 46 sec latency 13.230 ms oleg117-client.virtnet: 1 17868 2.45 MB/sec warmup 47 sec latency 9.124 ms oleg117-client.virtnet: 1 18429 2.43 MB/sec warmup 48 sec latency 10.611 ms oleg117-client.virtnet: 1 19023 2.42 MB/sec warmup 49 sec latency 9.812 ms oleg117-client.virtnet: 1 19831 2.44 MB/sec warmup 50 sec latency 10.878 ms oleg117-client.virtnet: 1 20213 2.43 MB/sec warmup 51 sec latency 413.719 ms oleg117-client.virtnet: 1 20213 2.38 MB/sec warmup 52 sec latency 1413.838 ms oleg117-client.virtnet: 1 20213 2.33 MB/sec warmup 53 sec latency 2414.005 ms oleg117-client.virtnet: 1 20213 2.29 MB/sec warmup 54 sec latency 3414.207 ms oleg117-client.virtnet: 1 20989 2.36 MB/sec warmup 55 sec latency 3840.468 ms oleg117-client.virtnet: 1 21780 2.42 MB/sec warmup 56 sec latency 10.636 ms oleg117-client.virtnet: 1 22500 2.42 MB/sec warmup 57 sec latency 9.281 ms oleg117-client.virtnet: 1 23280 2.43 MB/sec warmup 58 sec latency 8.974 ms oleg117-client.virtnet: 1 24301 2.51 MB/sec warmup 59 sec latency 8.030 ms oleg117-client.virtnet: 1 25947 1.95 MB/sec execute 1 sec latency 12.261 ms oleg117-client.virtnet: 1 26656 2.67 MB/sec execute 2 sec latency 90.827 ms oleg117-client.virtnet: 1 26656 1.78 MB/sec execute 3 sec latency 1090.988 ms oleg117-client.virtnet: 1 26656 1.33 MB/sec execute 4 sec latency 2091.192 ms oleg117-client.virtnet: 1 26656 1.07 MB/sec execute 5 sec latency 3091.368 ms oleg117-client.virtnet: 1 26656 0.89 MB/sec execute 6 sec latency 4091.533 ms oleg117-client.virtnet: 1 26656 0.76 MB/sec execute 7 sec latency 5091.692 ms oleg117-client.virtnet: 1 26656 0.67 MB/sec execute 8 sec latency 6091.849 ms oleg117-client.virtnet: 1 26656 0.59 MB/sec execute 9 sec latency 7092.042 ms oleg117-client.virtnet: 1 26656 0.53 MB/sec execute 10 sec latency 8092.229 ms oleg117-client.virtnet: 1 26656 0.48 MB/sec execute 11 sec latency 9092.404 ms oleg117-client.virtnet: 1 26656 0.44 MB/sec execute 12 sec latency 10092.578 ms oleg117-client.virtnet: 1 26656 0.41 MB/sec execute 13 sec latency 11092.702 ms oleg117-client.virtnet: 1 26656 0.38 MB/sec execute 14 sec latency 12092.841 ms oleg117-client.virtnet: 1 26656 0.36 MB/sec execute 15 sec latency 13093.015 ms oleg117-client.virtnet: 1 26656 0.33 MB/sec execute 16 sec latency 14093.216 ms oleg117-client.virtnet: 1 26656 0.31 MB/sec execute 17 sec latency 15093.392 ms oleg117-client.virtnet: 1 26656 0.30 MB/sec execute 18 sec latency 16093.556 ms oleg117-client.virtnet: 1 26656 0.28 MB/sec executoleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12936 3593224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36528 3569736 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49464 7162960 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36408 3567680 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7160764 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 e 19 sec latency 17093.716 ms oleg117-client.virtnet: 1 26656 0.27 MB/sec execute 20 sec latency 18093.848 ms oleg117-client.virtnet: 1 26656 0.25 MB/sec execute 21 sec latency 19093.969 ms oleg117-client.virtnet: 1 26656 0.24 MB/sec execute 22 sec latency 20094.157 ms oleg117-client.virtnet: 1 26767 0.24 MB/sec execute 23 sec latency 20899.694 ms oleg117-client.virtnet: 1 27797 0.52 MB/sec execute 24 sec latency 7.502 ms oleg117-client.virtnet: 1 28785 0.76 MB/sec execute 25 sec latency 9.518 ms oleg117-client.virtnet: 1 29461 0.81 MB/sec execute 26 sec latency 10.597 ms oleg117-client.virtnet: 1 30266 0.91 MB/sec execute 27 sec latency 7.929 ms oleg117-client.virtnet: 1 31345 1.13 MB/sec execute 28 sec latency 6.874 ms oleg117-client.virtnet: 1 32324 1.32 MB/sec execute 29 sec latency 8.737 ms oleg117-client.virtnet: 1 32964 1.31 MB/sec execute 30 sec latency 9.530 ms oleg117-client.virtnet: 1 33774 1.41 MB/sec execute 31 sec latency 8.744 ms oleg117-client.virtnet: 1 34860 1.57 MB/sec execute 32 sec latency 7.150 ms oleg117-client.virtnet: 1 35790 1.73 MB/sec execute 33 sec latency 9.521 ms oleg117-client.virtnet: 1 36411 1.71 MB/sec execute 34 sec latency 9.572 ms oleg117-client.virtnet: 1 37130 1.79 MB/sec execute 35 sec latency 9.281 ms oleg117-client.virtnet: 1 37945 1.79 MB/sec execute 36 sec latency 10.333 ms oleg117-client.virtnet: 1 39082 2.02 MB/sec execute 37 sec latency 13.783 ms oleg117-client.virtnet: 1 39544 2.01 MB/sec execute 38 sec latency 12.489 ms oleg117-client.virtnet: 1 40187 2.01 MB/sec execute 39 sec latency 10.056 ms oleg117-client.virtnet: 1 40930 2.05 MB/sec execute 40 sec latency 9.964 ms oleg117-client.virtnet: 1 41823 2.13 MB/sec execute 41 sec latency 7.416 ms oleg117-client.virtnet: 1 42802 2.27 MB/sec execute 42 sec latency 10.013 ms oleg117-client.virtnet: 1 43287 2.23 MB/sec execute 43 sec latency 11.285 ms oleg117-client.virtnet: 1 43861 2.22 MB/sec execute 44 sec latency 11.001 ms oleg117-client.virtnet: 1 44553 2.25 MB/sec execute 45 sec latency 87.483 ms oleg117-client.virtnet: 1 44553 2.20 MB/sec execute 46 sec latency 1087.632 ms oleg117-client.virtnet: 1 44553 2.15 MB/sec execute 47 sec latency 2087.822 ms oleg117-client.virtnet: 1 44553 2.11 MB/sec execute 48 sec latency 3088.020 ms oleg117-client.virtnet: 1 44708 2.07 MB/sec execute 49 sec latency 3844.747 ms oleg117-client.virtnet: 1 45475 2.14 MB/sec execute 50 sec latency 10.686 ms oleg117-client.virtnet: 1 46276 2.22 MB/sec execute 51 sec latency 14.005 ms oleg117-client.virtnet: 1 46768 2.21 MB/sec execute 52 sec latency 12.995 ms oleg117-client.virtnet: 1 47355 2.20 MB/sec execute 53 sec latency 12.103 ms oleg117-client.virtnet: 1 48010 2.22 MB/sec execute 54 sec latency 9.830 ms oleg117-client.virtnet: 1 48625 2.23 MB/sec execute 55 sec latency 8.983 ms oleg117-client.virtnet: 1 49707 2.36 MB/sec execute 56 sec latency 9.987 ms oleg117-client.virtnet: 1 50102 2.35 MB/sec execute 57 sec latency 193.584 ms oleg117-client.virtnet: 1 50102 2.31 MB/sec execute 58 sec latency 1193.747 ms oleg117-client.virtnet: 1 50102 2.27 MB/sec execute 59 sec latency 2193.965 ms oleg117-client.virtnet: 1 50102 2.23 MB/sec execute 60 sec latency 3194.171 ms oleg117-client.virtnet: 1 50102 2.19 MB/sec execute 61 sec latency 4194.348 ms oleg117-client.virtnet: 1 50102 2.16 MB/sec execute 62 sec latency 5194.509 ms oleg117-client.virtnet: 1 50102 2.12 MB/sec execute 63 sec latency 6195.661 ms oleg117-client.virtnet: 1 50102 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12812 3592936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36404 3567340 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49216 7160276 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12932 3593096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36408 3567680 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49340 7160776 1% /mnt/lustre 2.09 MB/sec execute 64 sec latency 7195.832 ms oleg117-client.virtnet: 1 50102 2.06 MB/sec execute 65 sec latency 8196.017 ms oleg117-client.virtnet: 1 50102 2.03 MB/sec execute 66 sec latency 9196.207 ms oleg117-client.virtnet: 1 50102 2.00 MB/sec execute 67 sec latency 10196.397 ms oleg117-client.virtnet: 1 50102 1.97 MB/sec execute 68 sec latency 11196.541 ms oleg117-client.virtnet: 1 50102 1.94 MB/sec execute 69 sec latency 12196.701 ms oleg117-client.virtnet: 1 50102 1.91 MB/sec execute 70 sec latency 13196.883 ms oleg117-client.virtnet: 1 50102 1.88 MB/sec execute 71 sec latency 14197.067 ms oleg117-client.virtnet: 1 50102 1.86 MB/sec execute 72 sec latency 15197.213 ms oleg117-client.virtnet: 1 50102 1.83 MB/sec execute 73 sec latency 16197.367 ms oleg117-client.virtnet: 1 50102 1.81 MB/sec execute 74 sec latency 17197.595 ms oleg117-client.virtnet: 1 50102 1.78 MB/sec execute 75 sec latency 18197.794 ms oleg117-client.virtnet: 1 50102 1.76 MB/sec execute 76 sec latency 19198.320 ms oleg117-client.virtnet: 1 50102 1.74 MB/sec execute 77 sec latency 20198.446 ms oleg117-client.virtnet: 1 50480 1.72 MB/sec execute 78 sec latency 20398.403 ms oleg117-client.virtnet: 1 51015 1.73 MB/sec execute 79 sec latency 12.279 ms oleg117-client.virtnet: 1 51651 1.74 MB/sec execute 80 sec latency 10.538 ms oleg117-client.virtnet: 1 52440 1.79 MB/sec execute 81 sec latency 8.561 ms oleg117-client.virtnet: 1 53380 1.85 MB/sec execute 82 sec latency 12.235 ms oleg117-client.virtnet: 1 53926 1.85 MB/sec execute 83 sec latency 11.129 ms oleg117-client.virtnet: 1 54621 1.88 MB/sec execute 84 sec latency 10.943 ms oleg117-client.virtnet: 1 55299 1.87 MB/sec execute 85 sec latency 8.600 ms oleg117-client.virtnet: 1 56442 1.93 MB/sec execute 86 sec latency 7.609 ms oleg117-client.virtnet: 1 57246 1.98 MB/sec execute 87 sec latency 8.103 ms oleg117-client.virtnet: 1 57959 1.98 MB/sec execute 88 sec latency 10.571 ms oleg117-client.virtnet: 1 58740 1.99 MB/sec execute 89 sec latency 8.417 ms oleg117-client.virtnet: 1 59781 2.05 MB/sec execute 90 sec latency 7.216 ms oleg117-client.virtnet: 1 60725 2.10 MB/sec execute 91 sec latency 12.253 ms oleg117-client.virtnet: 1 61307 2.09 MB/sec execute 92 sec latency 11.464 ms oleg117-client.virtnet: 1 62010 2.11 MB/sec execute 93 sec latency 9.958 ms oleg117-client.virtnet: 1 62878 2.12 MB/sec execute 94 sec latency 8.293 ms oleg117-client.virtnet: 1 64012 2.20 MB/sec execute 95 sec latency 8.261 ms oleg117-client.virtnet: 1 64560 2.19 MB/sec execute 96 sec latency 14.727 ms oleg117-client.virtnet: 1 65295 2.22 MB/sec execute 97 sec latency 10.605 ms oleg117-client.virtnet: 1 66078 2.21 MB/sec execute 98 sec latency 8.777 ms oleg117-client.virtnet: 1 67227 2.27 MB/sec execute 99 sec latency 8.939 ms oleg117-client.virtnet: 1 67842 2.29 MB/sec execute 100 sec latency 106.951 ms oleg117-client.virtnet: 1 67842 2.27 MB/sec execute 101 sec latency 1107.108 ms oleg117-client.virtnet: 1 67842 2.25 MB/sec execute 102 sec latency 2107.286 ms oleg117-client.virtnet: 1 67842 2.23 MB/sec execute 103 sec latency 3107.467 ms oleg117-client.virtnet: 1 68170 2.21 MB/sec execute 104 sec latency 3566.551 ms oleg117-client.virtnet: 1 68837 2.23 MB/sec execute 105 sec latency 10.376 ms oleg117-client.virtnet: 1 69583 2.22 MB/sec execute 106 sec latency 9.961 ms oleg117-client.virtnet: 1 70776 2.28 MB/sec execute 107 sec latency 7.581 ms oleg117-client.virtnet: 1 71511 2.30 MB/sec execute 108 sec latency 7.712 ms oleg117test_70b fail mds1 7 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13204 3593044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36236 3570392 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49440 7163436 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 -client.virtnet: 1 71952 2.29 MB/sec execute 109 sec latency 371.493 ms oleg117-client.virtnet: 1 72783 2.31 MB/sec execute 110 sec latency 415.622 ms oleg117-client.virtnet: 1 73767 2.34 MB/sec execute 111 sec latency 6.616 ms oleg117-client.virtnet: 1 73911 2.33 MB/sec execute 112 sec latency 918.816 ms oleg117-client.virtnet: 1 73911 2.31 MB/sec execute 113 sec latency 1919.022 ms oleg117-client.virtnet: 1 73911 2.29 MB/sec execute 114 sec latency 2919.210 ms oleg117-client.virtnet: 1 73911 2.27 MB/sec execute 115 sec latency 3919.399 ms oleg117-client.virtnet: 1 73911 2.25 MB/sec execute 116 sec latency 4919.557 ms oleg117-client.virtnet: 1 73911 2.23 MB/sec execute 117 sec latency 5919.723 ms oleg117-client.virtnet: 1 73911 2.21 MB/sec execute 118 sec latency 6919.946 ms oleg117-client.virtnet: 1 73911 2.19 MB/sec execute 119 sec latency 7920.163 ms oleg117-client.virtnet: 1 73911 2.18 MB/sec execute 120 sec latency 8920.376 ms oleg117-client.virtnet: 1 73911 2.16 MB/sec execute 121 sec latency 9920.530 ms oleg117-client.virtnet: 1 73911 2.14 MB/sec execute 122 sec latency 10920.706 ms oleg117-client.virtnet: 1 73911 2.12 MB/sec execute 123 sec latency 11920.972 ms oleg117-client.virtnet: 1 73911 2.11 MB/sec execute 124 sec latency 12921.127 ms oleg117-client.virtnet: 1 73911 2.09 MB/sec execute 125 sec latency 13921.303 ms oleg117-client.virtnet: 1 73911 2.07 MB/sec execute 126 sec latency 14921.486 ms oleg117-client.virtnet: 1 73911 2.06 MB/sec execute 127 sec latency 15921.670 ms oleg117-client.virtnet: 1 73911 2.04 MB/sec execute 128 sec latency 16921.922 ms oleg117-client.virtnet: 1 73911 2.02 MB/sec execute 129 sec latency 17922.103 ms oleg117-client.virtnet: 1 73911 2.01 MB/sec execute 130 sec latency 18922.300 ms oleg117-client.virtnet: 1 73911 1.99 MB/sec execute 131 sec latency 19922.495 ms oleg117-client.virtnet: 1 74164 1.98 MB/sec execute 132 sec latency 20709.715 ms oleg117-client.virtnet: 1 74883 2.01 MB/sec execute 133 sec latency 9.989 ms oleg117-client.virtnet: 1 75436 2.00 MB/sec execute 134 sec latency 13.442 ms oleg117-client.virtnet: 1 76131 2.02 MB/sec execute 135 sec latency 9.237 ms oleg117-client.virtnet: 1 76887 2.02 MB/sec execute 136 sec latency 9.415 ms oleg117-client.virtnet: 1 77870 2.06 MB/sec execute 137 sec latency 9.641 ms oleg117-client.virtnet: 1 78497 2.08 MB/sec execute 138 sec latency 12.068 ms oleg117-client.virtnet: 1 79067 2.07 MB/sec execute 139 sec latency 15.216 ms oleg117-client.virtnet: 1 79798 2.09 MB/sec execute 140 sec latency 9.228 ms oleg117-client.virtnet: 1 80542 2.09 MB/sec execute 141 sec latency 8.062 ms oleg117-client.virtnet: 1 81700 2.14 MB/sec execute 142 sec latency 9.287 ms oleg117-client.virtnet: 1 82214 2.14 MB/sec execute 143 sec latency 9.689 ms oleg117-client.virtnet: 1 82866 2.14 MB/sec execute 144 sec latency 10.544 ms oleg117-client.virtnet: 1 83626 2.15 MB/sec execute 145 sec latency 9.133 ms oleg117-client.virtnet: 1 84535 2.18 MB/sec execute 146 sec latency 8.230 ms oleg117-client.virtnet: 1 85449 2.21 MB/sec execute 147 sec latency 9.503 ms oleg117-client.virtnet: 1 85979 2.20 MB/sec execute 148 sec latency 11.481 ms oleg117-client.virtnet: 1 86676 2.22 MB/sec execute 149 sec latency 8.667 ms oleg117-client.virtnet: 1 87320 2.21 MB/sec execute 150 sec latency 10.839 ms oleg117-client.virtnet: 1 88168 2.24 MB/sec execute 151 sec latency 9.249 ms oleg117-client.virtnet: 1 89030 2.26 MB/sec execute 152 sec latency 9.252 ms oleg117-client.virtnet: 1 89599 2.25 MB/sec execoleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13204 3592928 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36236 3570272 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49440 7163200 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12868 3593136 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3569812 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49264 7162948 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 ute 153 sec latency 13.998 ms oleg117-client.virtnet: 1 90238 2.27 MB/sec execute 154 sec latency 11.477 ms oleg117-client.virtnet: 1 90451 2.25 MB/sec execute 155 sec latency 631.759 ms oleg117-client.virtnet: 1 90451 2.24 MB/sec execute 156 sec latency 1631.884 ms oleg117-client.virtnet: 1 90451 2.23 MB/sec execute 157 sec latency 2632.058 ms oleg117-client.virtnet: 1 90451 2.21 MB/sec execute 158 sec latency 3632.278 ms oleg117-client.virtnet: 1 90932 2.20 MB/sec execute 159 sec latency 4001.657 ms oleg117-client.virtnet: 1 91933 2.23 MB/sec execute 160 sec latency 9.250 ms oleg117-client.virtnet: 1 92658 2.26 MB/sec execute 161 sec latency 9.828 ms oleg117-client.virtnet: 1 93250 2.25 MB/sec execute 162 sec latency 10.094 ms oleg117-client.virtnet: 1 93930 2.26 MB/sec execute 163 sec latency 10.101 ms oleg117-client.virtnet: 1 94617 2.26 MB/sec execute 164 sec latency 59.220 ms oleg117-client.virtnet: 1 95179 2.28 MB/sec execute 165 sec latency 307.965 ms oleg117-client.virtnet: 1 96043 2.30 MB/sec execute 166 sec latency 11.768 ms oleg117-client.virtnet: 1 96241 2.29 MB/sec execute 167 sec latency 535.593 ms oleg117-client.virtnet: 1 96241 2.28 MB/sec execute 168 sec latency 1535.791 ms oleg117-client.virtnet: 1 96241 2.26 MB/sec execute 169 sec latency 2535.976 ms oleg117-client.virtnet: 1 96241 2.25 MB/sec execute 170 sec latency 3536.150 ms oleg117-client.virtnet: 1 96241 2.24 MB/sec execute 171 sec latency 4536.388 ms oleg117-client.virtnet: 1 96241 2.22 MB/sec execute 172 sec latency 5536.568 ms oleg117-client.virtnet: 1 96241 2.21 MB/sec execute 173 sec latency 6536.764 ms oleg117-client.virtnet: 1 96241 2.20 MB/sec execute 174 sec latency 7536.961 ms oleg117-client.virtnet: 1 96241 2.19 MB/sec execute 175 sec latency 8537.147 ms oleg117-client.virtnet: 1 96241 2.17 MB/sec execute 176 sec latency 9537.372 ms oleg117-client.virtnet: 1 96241 2.16 MB/sec execute 177 sec latency 10537.558 ms oleg117-client.virtnet: 1 96241 2.15 MB/sec execute 178 sec latency 11537.743 ms oleg117-client.virtnet: 1 96241 2.14 MB/sec execute 179 sec latency 12537.892 ms oleg117-client.virtnet: 1 96241 2.12 MB/sec execute 180 sec latency 13538.057 ms oleg117-client.virtnet: 1 96241 2.11 MB/sec execute 181 sec latency 14538.287 ms oleg117-client.virtnet: 1 96241 2.10 MB/sec execute 182 sec latency 15538.479 ms oleg117-client.virtnet: 1 96241 2.09 MB/sec execute 183 sec latency 16538.661 ms oleg117-client.virtnet: 1 96241 2.08 MB/sec execute 184 sec latency 17538.853 ms oleg117-client.virtnet: 1 96241 2.07 MB/sec execute 185 sec latency 18539.038 ms oleg117-client.virtnet: 1 96241 2.06 MB/sec execute 186 sec latency 19539.279 ms oleg117-client.virtnet: 1 96407 2.05 MB/sec execute 187 sec latency 20235.278 ms oleg117-client.virtnet: 1 97119 2.05 MB/sec execute 188 sec latency 8.211 ms oleg117-client.virtnet: 1 97927 2.05 MB/sec execute 189 sec latency 8.615 ms oleg117-client.virtnet: 1 99004 2.08 MB/sec execute 190 sec latency 8.629 ms oleg117-client.virtnet: 1 99757 2.10 MB/sec execute 191 sec latency 9.106 ms oleg117-client.virtnet: 1 100433 2.10 MB/sec execute 192 sec latency 8.113 ms oleg117-client.virtnet: 1 101199 2.11 MB/sec execute 193 sec latency 9.088 ms oleg117-client.virtnet: 1 102123 2.13 MB/sec execute 194 sec latency 11.417 ms oleg117-client.virtnet: 1 103168 2.16 MB/sec execute 195 sec latency 8.548 ms oleg117-client.virtnet: 1 103797 2.15 MB/sec execute 196 sec latency 10.472 ms oleg117-client.virtnet: 1 104571 2.16 MB/sec execute 197 sec latency 8.542 ms oleg117-cFailover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36420 3567540 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49364 7160624 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 lient.virtnet: 1 105379 2.16 MB/sec execute 198 sec latency 7.861 ms oleg117-client.virtnet: 1 106592 2.20 MB/sec execute 199 sec latency 7.773 ms oleg117-client.virtnet: 1 107202 2.20 MB/sec execute 200 sec latency 11.554 ms oleg117-client.virtnet: 1 107969 2.21 MB/sec execute 201 sec latency 8.739 ms oleg117-client.virtnet: 1 108802 2.21 MB/sec execute 202 sec latency 8.994 ms oleg117-client.virtnet: 1 109980 2.25 MB/sec execute 203 sec latency 8.060 ms oleg117-client.virtnet: 1 110594 2.25 MB/sec execute 204 sec latency 8.029 ms oleg117-client.virtnet: 1 111408 2.26 MB/sec execute 205 sec latency 9.819 ms oleg117-client.virtnet: 1 112184 2.26 MB/sec execute 206 sec latency 8.127 ms oleg117-client.virtnet: 1 113300 2.29 MB/sec execute 207 sec latency 8.523 ms oleg117-client.virtnet: 1 113991 2.30 MB/sec execute 208 sec latency 8.652 ms oleg117-client.virtnet: 1 114698 2.30 MB/sec execute 209 sec latency 12.206 ms oleg117-client.virtnet: 1 115012 2.30 MB/sec execute 210 sec latency 653.303 ms oleg117-client.virtnet: 1 115012 2.29 MB/sec execute 211 sec latency 1653.554 ms oleg117-client.virtnet: 1 115012 2.28 MB/sec execute 212 sec latency 2653.738 ms oleg117-client.virtnet: 1 115247 2.27 MB/sec execute 213 sec latency 3298.263 ms oleg117-client.virtnet: 1 116034 2.27 MB/sec execute 214 sec latency 7.681 ms oleg117-client.virtnet: 1 117210 2.31 MB/sec execute 215 sec latency 10.691 ms oleg117-client.virtnet: 1 117780 2.30 MB/sec execute 216 sec latency 11.020 ms oleg117-client.virtnet: 1 118498 2.31 MB/sec execute 217 sec latency 10.490 ms oleg117-client.virtnet: 1 119253 2.31 MB/sec execute 218 sec latency 8.983 ms oleg117-client.virtnet: 1 120298 2.33 MB/sec execute 219 sec latency 14.664 ms oleg117-client.virtnet: 1 121063 2.35 MB/sec execute 220 sec latency 8.408 ms oleg117-client.virtnet: 1 121261 2.34 MB/sec execute 221 sec latency 720.250 ms oleg117-client.virtnet: 1 121261 2.33 MB/sec execute 222 sec latency 1720.421 ms oleg117-client.virtnet: 1 121261 2.32 MB/sec execute 223 sec latency 2720.586 ms oleg117-client.virtnet: 1 121261 2.30 MB/sec execute 224 sec latency 3720.756 ms oleg117-client.virtnet: 1 121261 2.29 MB/sec execute 225 sec latency 4720.970 ms oleg117-client.virtnet: 1 121261 2.28 MB/sec execute 226 sec latency 5721.185 ms oleg117-client.virtnet: 1 121261 2.27 MB/sec execute 227 sec latency 6721.362 ms oleg117-client.virtnet: 1 121261 2.26 MB/sec execute 228 sec latency 7721.561 ms oleg117-client.virtnet: 1 121261 2.25 MB/sec execute 229 sec latency 8721.741 ms oleg117-client.virtnet: 1 121261 2.24 MB/sec execute 230 sec latency 9721.967 ms oleg117-client.virtnet: 1 121261 2.24 MB/sec execute 231 sec latency 10722.126 ms oleg117-client.virtnet: 1 121261 2.23 MB/sec execute 232 sec latency 11722.253 ms oleg117-client.virtnet: 1 121261 2.22 MB/sec execute 233 sec latency 12722.384 ms oleg117-client.virtnet: 1 121261 2.21 MB/sec execute 234 sec latency 13722.569 ms oleg117-client.virtnet: 1 121261 2.20 MB/sec execute 235 sec latency 14722.742 ms oleg117-client.virtnet: 1 121261 2.19 MB/sec execute 236 sec latency 15722.941 ms oleg117-client.virtnet: 1 121261 2.18 MB/sec execute 237 sec latency 16723.136 ms oleg117-client.virtnet: 1 121261 2.17 MB/sec execute 238 sec latency 17723.321 ms oleg117-client.virtnet: 1 121261 2.16 MB/sec execute 239 sec latency 18723.516 ms oleg117-client.virtnet: 1 121261 2.15 MB/sec execute 240 sec latency 19723.701 ms oleg117-client.virtnet: 1 121261 2.14 MB/sec execute 241 sec latency 20723.830 ms oleg117-client.virtnet: 1 121804 2.14 MB/sec oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 242 sec latency 20784.562 ms oleg117-client.virtnet: 1 122473 2.15 MB/sec execute 243 sec latency 10.133 ms oleg117-client.virtnet: 1 123123 2.15 MB/sec execute 244 sec latency 11.092 ms oleg117-client.virtnet: 1 124249 2.18 MB/sec execute 245 sec latency 10.038 ms oleg117-client.virtnet: 1 124843 2.18 MB/sec execute 246 sec latency 10.294 ms oleg117-client.virtnet: 1 125607 2.19 MB/sec execute 247 sec latency 11.083 ms oleg117-client.virtnet: 1 126379 2.18 MB/sec execute 248 sec latency 8.762 ms oleg117-client.virtnet: 1 127460 2.21 MB/sec execute 249 sec latency 8.304 ms oleg117-client.virtnet: 1 128194 2.22 MB/sec execute 250 sec latency 8.559 ms oleg117-client.virtnet: 1 128897 2.22 MB/sec execute 251 sec latency 11.145 ms oleg117-client.virtnet: 1 129644 2.22 MB/sec execute 252 sec latency 8.941 ms oleg117-client.virtnet: 1 130657 2.24 MB/sec execute 253 sec latency 7.706 ms oleg117-client.virtnet: 1 131578 2.26 MB/sec execute 254 sec latency 9.649 ms oleg117-client.virtnet: 1 132183 2.25 MB/sec execute 255 sec latency 13.339 ms oleg117-client.virtnet: 1 132925 2.26 MB/sec execute 256 sec latency 9.623 ms oleg117-client.virtnet: 1 133648 2.26 MB/sec execute 257 sec latency 12.150 ms oleg117-client.virtnet: 1 134831 2.29 MB/sec execute 258 sec latency 9.339 ms oleg117-client.virtnet: 1 135482 2.29 MB/sec execute 259 sec latency 13.528 ms oleg117-client.virtnet: 1 136273 2.30 MB/sec execute 260 sec latency 12.866 ms oleg117-client.virtnet: 1 137087 2.29 MB/sec execute 261 sec latency 9.404 ms oleg117-client.virtnet: 1 138184 2.32 MB/sec execute 262 sec latency 13.235 ms oleg117-client.virtnet: 1 138970 2.33 MB/sec execute 263 sec latency 7.280 ms oleg117-client.virtnet: 1 139833 2.34 MB/sec execute 264 sec latency 8.497 ms oleg117-client.virtnet: 1 140746 2.34 MB/sec execute 265 sec latency 7.597 ms oleg117-client.virtnet: 1 141932 2.37 MB/sec execute 266 sec latency 7.589 ms oleg117-client.virtnet: 1 142611 2.36 MB/sec execute 267 sec latency 10.154 ms oleg117-client.virtnet: 1 143497 2.37 MB/sec execute 268 sec latency 8.401 ms oleg117-client.virtnet: 1 144385 2.37 MB/sec execute 269 sec latency 7.577 ms oleg117-client.virtnet: 1 145541 2.40 MB/sec execute 270 sec latency 8.723 ms oleg117-client.virtnet: 1 146127 2.40 MB/sec execute 271 sec latency 9.335 ms oleg117-client.virtnet: 1 146723 2.40 MB/sec execute 272 sec latency 12.319 ms oleg117-client.virtnet: 1 147397 2.40 MB/sec execute 273 sec latency 12.857 ms oleg117-client.virtnet: 1 148072 2.40 MB/sec execute 274 sec latency 9.983 ms oleg117-client.virtnet: 1 149124 2.43 MB/sec execute 275 sec latency 10.308 ms oleg117-client.virtnet: 1 149655 2.43 MB/sec execute 276 sec latency 10.153 ms oleg117-client.virtnet: 1 150184 2.42 MB/sec execute 277 sec latency 13.932 ms oleg117-client.virtnet: 1 150826 2.43 MB/sec execute 278 sec latency 10.600 ms oleg117-client.virtnet: 1 151475 2.43 MB/sec execute 279 sec latency 8.215 ms oleg117-client.virtnet: 1 152547 2.45 MB/sec execute 280 sec latency 10.933 ms oleg117-client.virtnet: 1 153060 2.45 MB/sec execute 281 sec latency 9.334 ms oleg117-client.virtnet: 1 153738 2.45 MB/sec execute 282 sec latency 11.865 ms oleg117-client.virtnet: 1 154442 2.45 MB/sec execute 283 sec latency 9.671 ms oleg117-client.virtnet: 1 155332 2.46 MB/sec execute 284 sec latency 8.112 ms oleg117-client.virtnet: 1 156327 2.48 MB/sec execute 285 sec latency 9.490 ms oleg117-client.virtnet: 1 156907 2.48 MB/sec execute 286 sec latency 9.595 ms oleg117-client.virtnet: 1 157686 2.48 MB/sec execute 287 sec latency 7.847 ms oleg117-client.virtnet: 1 158501 2.48 MB/sec execute 288 sec latency 8.285 ms oleg117-client.virtnet: 1 159613 2.51 MB/sec execute 289 sec latency 9.311 ms oleg117-client.virtnet: 1 160160 2.51 MB/sec execute 290 sec latency 9.068 ms oleg117-client.virtnet: 1 160831 2.50 MB/sec execute 291 sec latency 14.778 ms oleg117-client.virtnet: 1 161589 2.51 MB/sec execute 292 sec latency 8.810 ms oleg117-client.virtnet: 1 162526 2.52 MB/sec execute 293 sec latency 7.947 ms oleg117-client.virtnet: 1 163464 2.54 MB/sec execute 294 sec latency 9.444 ms oleg117-client.virtnet: 1 164092 2.53 MB/sec execute 295 sec latency 13.133 ms oleg117-client.virtnet: 1 164873 2.54 MB/sec execute 296 sec latency 8.767 ms oleg117-client.virtnet: 1 165645 2.54 MB/sec execute 297 sec latency 8.514 ms oleg117-client.virtnet: 1 166838 2.56 MB/sec execute 298 sec latency 8.329 ms oleg117-client.virtnet: 1 167443 2.56 MB/sec execute 299 sec latency 10.180 ms oleg117-client.virtnet: 1 cleanup 300 sec oleg117-client.virtnet: 0 cleanup 300 sec oleg117-client.virtnet: oleg117-client.virtnet: Operation Count AvgLat MaxLat oleg117-client.virtnet: ---------------------------------------- oleg117-client.virtnet: NTCreateX 24728 7.701 20899.680 oleg117-client.virtnet: Close 18154 1.941 20784.535 oleg117-client.virtnet: Rename 1050 6.073 13.986 oleg117-client.virtnet: Unlink 5023 2.272 3844.731 oleg117-client.virtnet: Qpathinfo 22432 1.202 3566.534 oleg117-client.virtnet: Qfileinfo 3900 0.112 1.761 oleg117-client.virtnet: Qfsinfo 4116 0.162 2.988 oleg117-client.virtnet: Sfileinfo 2034 2.758 10.581 oleg117-client.virtnet: Find 8665 0.322 7.115 oleg117-client.virtnet: WriteX 12211 0.820 7.910 oleg117-client.virtnet: ReadX 38725 0.011 1.632 oleg117-client.virtnet: LockX 80 0.809 1.713 oleg117-client.virtnet: UnlockX 80 0.802 1.337 oleg117-client.virtnet: Flush 1744 4.937 415.609 oleg117-client.virtnet: oleg117-client.virtnet: Throughput 2.55875 MB/sec 1 clients 1 procs max_latency=20899.694 ms oleg117-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg117-client.virtnet at Wed Jan 15 19:19:15 EST 2025 with return code 0 oleg117-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg117-client.virtnet oleg117-client.virtnet: /mnt/lustre/d70b.replay-single/oleg117-client.virtnet /mnt/lustre/d70b.replay-single/oleg117-client.virtnet oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg117-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg117-client.virtnet: removed directory: 'clients/client0' oleg117-client.virtnet: removed directory: 'clients' oleg117-client.virtnet: removed 'client.txt' oleg117-client.virtnet: /mnt/lustre/d70b.replay-single/oleg117-client.virtnet oleg117-client.virtnet: dbench successfully finished PASS 70b (363s) == replay-single test 70c: tar 2mdts recovery ============ 19:19:17 (1736986757) Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar 1100 striped dir -i0 -c2 -H all_char /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4984 1282704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6692 1280996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9640 3568956 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3575908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11056 7144864 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4956 1282732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 7440 1280248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9700 3566128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5304 3576464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15004 7142592 1% /mnt/lustre tar: Removing leading `/' from member names test_70c fail mds1 2 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 tar: Removing leading `/' from hard link targets oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (353s) == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 19:25:10 (1736987110) Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started 4573 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3448 1284240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6464 1281224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3604516 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3600956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7205472 1% /mnt/lustre test_70d fail mds1 1 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4976 1282712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5524 1282164 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3604516 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3600956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7205472 1% /mnt/lustre test_70d fail mds2 2 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4260: 4573 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (318s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:30:28 (1736987428) debug=+ha Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started PID=2827 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4132 1283556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3820 1283868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3604516 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3600956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7205472 1% /mnt/lustre test_70e fail mds1 1 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2644 1285044 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3604516 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3600956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7205472 1% /mnt/lustre test_70e fail mds2 2 times Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (314s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:35:42 (1736987742) mount clients oleg117-client.virtnet ... Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6568 3600452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6536 3600484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 13104 7200936 1% /mnt/lustre ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3574348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6536 3590076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 7984 7164424 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3569180 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3569212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7138392 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... test_70f failing OST 3 times Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3600404 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3600436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7200840 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... test_70f failing OST 4 times Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 5544 3575492 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6536 3600484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 12080 7175976 1% /mnt/lustre ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... test_70f failing OST 5 times Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... ldlm.namespaces.MGC192.168.201.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a8d73000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a8d73000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg117-client.virtnet' ... PASS 70f (141s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:38:03 (1736987883) Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started 9703 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3320 1284368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3880 1283808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3480 1284208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4100 1283588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail mds1 mds2 1 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3564 1284124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3684 1284004 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail mds1 mds2 2 times Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4260: 9703 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (359s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 19:44:02 (1736988242) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3776 1283912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3444 1284244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (31s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 19:44:33 (1736988273) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7490 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1285508 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1285788 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (33s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 19:45:06 (1736988306) Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:) Stopping client oleg117-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg117-server Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (28s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:45:34 (1736988334) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2196 1285492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.04 seconds: 473.65 ops/second PASS 80a (29s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:46:03 (1736988363) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1936 1285752 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.07 seconds: 280.43 ops/second PASS 80b (36s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:46:39 (1736988399) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.04 seconds: 494.93 ops/second PASS 80c (54s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:47:33 (1736988453) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2296 1285392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2296 1285392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.04 seconds: 474.89 ops/second PASS 80d (51s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:48:24 (1736988504) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.07 seconds: 272.38 ops/second PASS 80e (45s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:49:09 (1736988549) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 364.92 ops/second PASS 80f (26s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:49:35 (1736988575) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 391.95 ops/second PASS 80g (56s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:50:31 (1736988631) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.08 seconds: 248.49 ops/second PASS 80h (51s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:51:22 (1736988682) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (31s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:51:53 (1736988713) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1980 1285708 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (29s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:52:23 (1736988743) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1944 1285744 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1944 1285744 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (56s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:53:18 (1736988798) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1984 1285704 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (66s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:54:24 (1736988864) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (40s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:55:04 (1736988904) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (27s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:55:31 (1736988931) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1952 1285736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (63s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:56:34 (1736988994) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1285508 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3605604 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (48s) == replay-single test 84a: stale open during export disconnect ========================================================== 19:57:22 (1736989042) fail_loc=0x80000144 total: 1 open/close in 0.00 seconds: 284.82 ops/second oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 PASS 84a (5s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:57:27 (1736989047) before recovery: unused locks count = 301 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (23s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:57:50 (1736989070) before recovery: unused locks count = 100 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (23s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:58:13 (1736989093) Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:) Stopping client oleg117-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) PASS 86 (11s) == replay-single test 87a: write replay ================== 19:58:24 (1736989104) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1285508 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1648 3605372 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3264 7210776 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.106949 s, 78.4 MB/s Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0347802 s, 241 MB/s PASS 87a (23s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:58:47 (1736989127) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1285508 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9840 3597180 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11456 7202584 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.140307 s, 59.8 MB/s 8+0 records in 8+0 records out 8 bytes (8 B) copied, 0.00155798 s, 5.1 kB/s Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes (72 B) copied, 0.00473138 s, 15.2 kB/s PASS 87b (25s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:59:12 (1736989152) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2184 1285504 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9844 3597176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11460 7202580 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2184 1285504 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9844 3597176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11460 7202580 1% /mnt/lustre before test: last_id = 18625, next_id = 18594 Creating to objid 18625 on ost lustre-OST0000... total: 33 open/close in 0.07 seconds: 457.67 ops/second total: 8 open/close in 0.02 seconds: 492.64 ops/second before recovery: last_id = 18657, next_id = 18634 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 18665, next_id = 18634 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0272021 s, 19.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0227561 s, 23.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0213658 s, 24.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0232218 s, 22.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0213117 s, 24.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0212518 s, 24.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0235052 s, 22.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0248064 s, 21.1 MB/s -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18594 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18595 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18596 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18597 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18598 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18599 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18600 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18601 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18602 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18603 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18604 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18605 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18606 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18607 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18608 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18609 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18610 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18611 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18612 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18613 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18614 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18615 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18616 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18617 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18618 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18619 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18620 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18621 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18622 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18623 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18624 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18625 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18626 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18627 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18628 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18629 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18630 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18631 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18632 -rw-r--r-- 1 root root 0 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18633 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18637 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18638 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18639 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18640 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18641 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18642 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18643 -rw-r--r-- 1 root root 524288 Jan 15 19:59 /mnt/lustre/d88.replay-single/f-18644 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0217437 s, 24.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.022329 s, 23.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0214995 s, 24.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.021761 s, 24.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0207314 s, 25.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0252064 s, 20.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0216992 s, 24.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0215574 s, 24.3 MB/s PASS 88 (42s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 19:59:54 (1736989194) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. Waiting for MDT destroys to complete 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0668987 s, 157 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7646580 free_after: 7646580 PASS 89 (115s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 20:01:49 (1736989309) Create the files Fail ost1 lustre-OST0000_UUID, display the list of affected files Stopping /mnt/lustre-ost1 (opts:) on oleg117-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Querying files on shutdown ost1: lfs find --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 18666 0x48ea 0 * /mnt/lustre/d90.replay-single/f0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 18667 0x48eb 0 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Failover ost1 to oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 90 (18s) == replay-single test 93a: replay + reconnect ============ 20:02:07 (1736989327) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00124166 s, 825 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (61s) == replay-single test 93b: replay + reconnect on mds ===== 20:03:08 (1736989388) total: 20 open/close in 0.09 seconds: 230.07 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (108s) == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 20:04:56 (1736989496) fail_loc=0x1701 Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000347f0:0x1:0x0] 1 [0x24000a812:0x1:0x0] total: 20 open/close in 0.06 seconds: 327.25 ops/second PASS 100a (23s) == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 20:05:19 (1736989519) fail_loc=0x119 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000347f0:0x4:0x0] 1 [0x24000a812:0x4:0x0] total: 20 open/close in 0.04 seconds: 470.03 ops/second PASS 100b (27s) == replay-single test 100c: DNE: create striped dir, fail MDT0 ========================================================== 20:05:46 (1736989546) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1285376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Stopping /mnt/lustre-mds2 (opts:) on oleg117-server Failover mds2 to oleg117-server Starting mds2: -o localrecov -o abort_recov_mdt /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 open(/mnt/lustre/d100c.replay-single/striped_dir/f-0) error: No such file or directory total: 0 open/close in 0.00 seconds: 0.00 ops/second Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 1 [0x24000afe0:0x3:0x0] 0 [0x2000347f2:0x3:0x0] total: 20 open/close in 0.04 seconds: 478.77 ops/second PASS 100c (38s) == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 20:06:24 (1736989584) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1285416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2000 1285688 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 first stat failed: 5 PASS 101 (31s) == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 20:06:55 (1736989615) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (20:06:56) ... fail_loc=0 done (20:07:03) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (9s) == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 20:07:04 (1736989624) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (20:07:05) ... fail_loc=0 done (20:07:12) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (9s) == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 20:07:13 (1736989633) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1285380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7193656 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (20:07:16) ... fail_loc=0 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (20:07:40) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (28s) == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 20:07:41 (1736989661) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (20:07:42) ... fail_loc=0 Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec done (20:08:05) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (25s) == replay-single test 103: Check otr_next_id overflow ==== 20:08:06 (1736989686) fail_loc=0x80000162 total: 30 open/close in 0.13 seconds: 224.38 ops/second Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (27s) == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 20:08:33 (1736989713) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7193656 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110a.replay-single/striped_dir has type dir OK PASS 110a (29s) == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 20:09:02 (1736989742) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194048 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110b.replay-single/striped_dir has type dir OK PASS 110b (91s) == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 20:10:33 (1736989833) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110c.replay-single/striped_dir has type dir OK PASS 110c (27s) == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 20:11:00 (1736989860) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1285408 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110d.replay-single/striped_dir has type dir OK PASS 110d (94s) == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 20:12:34 (1736989954) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2232 1285456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110e.replay-single/striped_dir has type dir OK PASS 110e (115s) SKIP: replay-single test_110f skipping excluded test 110f == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 20:14:30 (1736990070) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1285416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110g.replay-single/striped_dir has type dir OK PASS 110g (130s) == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 20:16:40 (1736990200) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2000 1285688 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111a.replay-single/striped_dir: No such file or directory PASS 111a (26s) == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 20:17:06 (1736990226) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2252 1285436 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111b.replay-single/striped_dir: No such file or directory PASS 111b (89s) == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 20:18:35 (1736990315) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2296 1285392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2004 1285684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111c.replay-single/striped_dir: No such file or directory PASS 111c (131s) == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 20:20:46 (1736990446) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2004 1285684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg117-client: oleg117-client: ssh exited with exit code 95 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111d.replay-single/striped_dir: No such file or directory PASS 111d (115s) == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 20:22:41 (1736990561) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2300 1285388 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2004 1285684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111e.replay-single/striped_dir: No such file or directory PASS 111e (48s) == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 20:23:29 (1736990609) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2000 1285688 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111f.replay-single/striped_dir: No such file or directory PASS 111f (47s) == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 20:24:16 (1736990656) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111g.replay-single/striped_dir: No such file or directory PASS 111g (48s) == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 20:25:04 (1736990704) SKIP: replay-single test_112a needs >= 4 MDTs SKIP 112a (0s) == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 20:25:04 (1736990704) SKIP: replay-single test_112b needs >= 4 MDTs SKIP 112b (1s) == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 20:25:05 (1736990705) SKIP: replay-single test_112c needs >= 4 MDTs SKIP 112c (1s) == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 20:25:06 (1736990706) SKIP: replay-single test_112d needs >= 4 MDTs SKIP 112d (1s) == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 20:25:07 (1736990707) SKIP: replay-single test_112e needs >= 4 MDTs SKIP 112e (1s) == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 20:25:08 (1736990708) SKIP: replay-single test_112f needs >= 4 MDTs SKIP 112f (0s) == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 20:25:08 (1736990708) SKIP: replay-single test_112g needs >= 4 MDTs SKIP 112g (1s) == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 20:25:09 (1736990709) SKIP: replay-single test_112h needs >= 4 MDTs SKIP 112h (1s) == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 20:25:10 (1736990710) SKIP: replay-single test_112i needs >= 4 MDTs SKIP 112i (1s) == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 20:25:11 (1736990711) SKIP: replay-single test_112j needs >= 4 MDTs SKIP 112j (0s) == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 20:25:12 (1736990712) SKIP: replay-single test_112k needs >= 4 MDTs SKIP 112k (1s) == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 20:25:12 (1736990712) SKIP: replay-single test_112l needs >= 4 MDTs SKIP 112l (1s) == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 20:25:13 (1736990713) SKIP: replay-single test_112m needs >= 4 MDTs SKIP 112m (1s) == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 20:25:14 (1736990714) SKIP: replay-single test_112n needs >= 4 MDTs SKIP 112n (0s) == replay-single test 115: failover for create/unlink striped directory ========================================================== 20:25:14 (1736990714) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2264 1285424 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre striped dir -i1 -c2 -H all_char /mnt/lustre/d115.replay-single/test_0 striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_1 striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_2 striped dir -i1 -c2 -H all_char /mnt/lustre/d115.replay-single/test_3 striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_4 Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2296 1285392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_0 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_1 striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_2 striped dir -i0 -c2 -H all_char /mnt/lustre/d115.replay-single/test_3 striped dir -i0 -c2 -H all_char /mnt/lustre/d115.replay-single/test_4 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 115 (59s) == replay-single test 116a: large update log master MDT recovery ========================================================== 20:26:13 (1736990773) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1285368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre fail_loc=0x80001702 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116a.replay-single/striped_dir has type dir OK PASS 116a (28s) == replay-single test 116b: large update log slave MDT recovery ========================================================== 20:26:41 (1736990801) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1285368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre fail_loc=0x80001702 Failing mds2 on oleg117-server Stopping /mnt/lustre-mds2 (opts:) on oleg117-server reboot facets: mds2 Failover mds2 to oleg117-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116b.replay-single/striped_dir has type dir OK PASS 116b (26s) == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 20:27:07 (1736990827) SKIP: replay-single test_117 needs >= 4 MDTs SKIP 117 (1s) == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 20:27:08 (1736990828) fail_loc=0x1705 lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error fail_loc=0x0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2200 1285488 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 118 (27s) == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 20:27:35 (1736990855) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2556 1285132 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server fail_loc=0x80000714 fail_val=65 Starting mds1: -o localrecov -o recovery_time_hard=60 /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 71 sec mdt.lustre-MDT0000.recovery_time_hard=180 PASS 119 (86s) == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 20:29:01 (1736990941) Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 first stat failed: 5 PASS 120 (24s) == replay-single test 121: lock replay timed out and race ========================================================== 20:29:25 (1736990965) multiop /mnt/lustre/f121.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7490 Stopping /mnt/lustre-mds1 (opts:) on oleg117-server Failover mds1 to oleg117-server fail_loc=0x721 fail_val=0 at_max=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 fail_loc=0x0 at_max=600 PASS 121 (192s) == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 20:32:37 (1736991157) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2668 1285020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2376 1285312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130a (29s) == replay-single test 130b: DoM file create (inherited) replay ========================================================== 20:33:06 (1736991186) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2672 1285016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2376 1285312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130b (28s) == replay-single test 131a: DoM file write lock replay === 20:33:34 (1736991214) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2680 1285008 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2376 1285312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre 1+0 records in 1+0 records out 8 bytes (8 B) copied, 0.00111096 s, 7.2 kB/s Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 131a (28s) SKIP: replay-single test_131b skipping excluded test 131b == replay-single test 132a: PFL new component instantiate replay ========================================================== 20:34:03 (1736991243) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2684 1285004 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2376 1285312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1616 3605404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194384 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168993 s, 62.0 MB/s /mnt/lustre/f132a.replay-single lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x4b62:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x4e2a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x4b63:0x0] } Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f132a.replay-single lcm_layout_gen: 4 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x4b62:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x4e2a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x4b63:0x0] } PASS 132a (28s) == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 20:34:31 (1736991271) seq.srv-lustre-MDT0001.space=clear Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre fail_val=700 fail_loc=0x80000123 Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 133 (26s) == replay-single test 134: replay creation of a file created in a pool ========================================================== 20:34:57 (1736991297) Creating new pool oleg117-server: Pool lustre.pool_134 created Adding targets to pool oleg117-server: OST lustre-OST0001_UUID added to pool lustre.pool_134 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2736 1284952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2640 3604380 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20680 7193360 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: pool_134 lustre.pool_134 oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.pool_134 oleg117-server: Pool lustre.pool_134 destroyed PASS 134 (35s) == replay-single test 135: Server failure in lock replay phase ========================================================== 20:35:32 (1736991332) Failing ost1 on oleg117-server Stopping /mnt/lustre-ost1 (opts:) on oleg117-server reboot facets: ost1 Failover ost1 to oleg117-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2712 1284976 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18040 3588980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2640 3604380 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20680 7193360 1% /mnt/lustre Stopping /mnt/lustre-ost1 (opts:) on oleg117-server Failover ost1 to oleg117-server oleg117-server: oleg117-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_val=20 fail_loc=0x32d Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:) on oleg117-server Failover ost1 to oleg117-server oleg117-server: oleg117-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 5 PASS 135 (113s) == replay-single test 202: pfl replay should recovery layout generation ========================================================== 20:37:25 (1736991445) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2716 1284972 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18116 3588904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2656 3604364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20772 7193268 1% /mnt/lustre Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server reboot facets: mds1 Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 202 (28s) == replay-single test complete, duration 8589 sec ======== 20:37:53 (1736991473)