-----============= acceptance-small: replay-single ============----- Fri Oct 24 08:55:15 EDT 2025 oleg335-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 oleg335-client.virtnet: executing check_config_client /mnt/lustre oleg335-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg335-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9412061b0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9412061b0800.idle_timeout=debug disable quota as required oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 osd-ldiskfs.track_declares_assert=1 == replay-single test 0a: empty replay =================== 08:55:39 (1761310539) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (45s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 08:56:24 (1761310584) Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.19 seconds: 107.32 ops/second - unlinked 0 (time 1761310612 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (30s) == replay-single test 0c: check replay-barrier =========== 08:56:54 (1761310614) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (106s) == replay-single test 0d: expired recovery with no clients ========================================================== 08:58:41 (1761310721) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre PASS 0d (99s) == replay-single test 1: simple create =================== 09:00:20 (1761310820) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (57s) == replay-single test 2a: touch ========================== 09:01:17 (1761310877) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (45s) == replay-single test 2b: touch ========================== 09:02:03 (1761310923) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (50s) == replay-single test 2c: setstripe replay =============== 09:02:53 (1761310973) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (56s) == replay-single test 2d: setdirstripe replay ============ 09:03:49 (1761311029) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (49s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 09:04:38 (1761311078) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (48s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 09:05:26 (1761311126) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (35s) == replay-single test 3b: replay failed open -ENOMEM ===== 09:06:01 (1761311161) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (45s) == replay-single test 3c: replay failed open -ENOMEM ===== 09:06:46 (1761311206) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (46s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 09:07:32 (1761311252) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (52s) == replay-single test 4b: |x| rm 10 files ================ 09:08:24 (1761311304) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (59s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 09:09:23 (1761311363) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605352 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7210984 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (90s) == replay-single test 6a: mkdir + contained create ======= 09:10:53 (1761311453) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2932 7211108 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (64s) == replay-single test 6b: |X| rmdir ====================== 09:11:57 (1761311517) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1848 1285840 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (52s) == replay-single test 7: mkdir |X| contained create ====== 09:12:49 (1761311569) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1848 1285840 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (61s) == replay-single test 8: creat open |X| close ============ 09:13:50 (1761311630) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (57s) == replay-single test 9: |X| create (same inum/gen) ====== 09:14:47 (1761311687) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (62s) == replay-single test 10: create |X| rename unlink ======= 09:15:50 (1761311750) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (58s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 09:16:48 (1761311808) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3605576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2832 7211208 1% /mnt/lustre new old Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (59s) == replay-single test 12: open, unlink |X| close ========= 09:17:46 (1761311866) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (56s) == replay-single test 13: open chmod 0 |x| write close === 09:18:43 (1761311923) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (55s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 09:19:39 (1761311979) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (58s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 09:20:37 (1761312037) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (55s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 09:21:32 (1761312092) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (55s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 09:22:27 (1761312147) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (55s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 09:23:22 (1761312202) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 pid: 53822 will close Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (115s) == replay-single test 19: mcreate, open, write, rename === 09:25:17 (1761312317) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3605572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2840 7211200 1% /mnt/lustre old Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (55s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:26:13 (1761312373) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605568 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2844 7211196 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (49s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 09:27:02 (1761312422) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1026 0x402 0 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown oleg335-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg335-client: oleg335-client: ssh exited with exit code 5 6630+0 records in 6629+0 records out 27152384 bytes (27 MB, 26 MiB) copied, 2.36744 s, 11.5 MB/s Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg335-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 2844, after 2844 PASS 20b (59s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 09:28:01 (1761312481) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 -rw-r--r-- 1 root root 1 Oct 24 09:28 /mnt/lustre/f20c.replay-single PASS 20c (6s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 09:28:07 (1761312487) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1392 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2844 7211136 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (51s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:28:58 (1761312538) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (56s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 09:29:54 (1761312594) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (54s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 09:30:48 (1761312648) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (52s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:31:40 (1761312700) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (51s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 09:32:31 (1761312751) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (48s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:33:19 (1761312799) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (50s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 09:34:10 (1761312850) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (48s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:34:58 (1761312898) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (50s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:35:48 (1761312948) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (52s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 09:36:40 (1761313000) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (49s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 09:37:29 (1761313049) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 oleg335-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg335-client: oleg335-client: ssh exited with exit code 5 PASS 32 (8s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 09:37:37 (1761313057) total: 10 open/close in 0.17 seconds: 57.39 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.15 seconds: 65.14 ops/second PASS 33a (38s) == replay-single test 33b: test fid seq allocation ======= 09:38:15 (1761313095) fail_loc=0x1311 total: 10 open/close in 0.15 seconds: 68.09 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.14 seconds: 72.93 ops/second PASS 33b (33s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 09:38:49 (1761313129) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (36s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 09:39:25 (1761313165) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (31s) == replay-single test 36: don't resend cancel ============ 09:39:55 (1761313195) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 36 (46s) == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 09:40:41 (1761313241) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (37s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:41:18 (1761313278) total: 800 open/close in 6.02 seconds: 132.82 ops/second - unlinked 0 (time 1761313288 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1952 1285736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761313341 ; total 0 ; last 0) total: 400 unlinks in 5 seconds: 80.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (75s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:42:33 (1761313353) total: 800 open/close in 8.30 seconds: 96.40 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1952 1285736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1452 3605508 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2848 7211132 1% /mnt/lustre - unlinked 0 (time 1761313373 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761313424 ; total 0 ; last 0) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (82s) == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 09:43:55 (1761313435) SKIP: replay-single test_40 layout_lock needs MDS connection for IO SKIP 40 (3s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:43:58 (1761313438) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0056428 s, 726 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150093 s, 273 kB/s PASS 41 (7s) == replay-single test 42: recovery after ost failure ===== 09:44:05 (1761313445) total: 800 open/close in 8.36 seconds: 95.68 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1761313464 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second debug=-1 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 wait for MDS to timeout and recover debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck - unlinked 0 (time 1761313538 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (100s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:45:45 (1761313545) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1285656 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (65s) == replay-single test 44a: race in target handle connect ========================================================== 09:46:50 (1761313610) at_max=40 1 of 10 (1761313614) service : cur 31 worst 31 (at 1761313285, 329s ago) 30 30 31 1 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 2 of 10 (1761313621) service : cur 31 worst 31 (at 1761313285, 336s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 3 of 10 (1761313628) service : cur 31 worst 31 (at 1761313285, 343s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 4 of 10 (1761313634) service : cur 31 worst 31 (at 1761313285, 350s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 5 of 10 (1761313641) service : cur 31 worst 31 (at 1761313285, 356s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 6 of 10 (1761313648) service : cur 31 worst 31 (at 1761313285, 363s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 7 of 10 (1761313654) service : cur 31 worst 31 (at 1761313285, 370s ago) 5 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 8 of 10 (1761313661) service : cur 31 worst 31 (at 1761313285, 376s ago) 6 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 9 of 10 (1761313668) service : cur 31 worst 31 (at 1761313285, 383s ago) 6 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre 10 of 10 (1761313674) service : cur 31 worst 31 (at 1761313285, 390s ago) 6 30 30 31 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (74s) == replay-single test 44b: race in target handle connect ========================================================== 09:48:04 (1761313684) 1 of 10 (1761313685) service : cur 31 worst 31 (at 1761313285, 401s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Connection timed out 2 of 10 (1761313692) service : cur 31 worst 31 (at 1761313285, 407s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 3 of 10 (1761313693) service : cur 31 worst 31 (at 1761313285, 408s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 4 of 10 (1761313694) service : cur 31 worst 31 (at 1761313285, 409s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 5 of 10 (1761313695) service : cur 31 worst 31 (at 1761313285, 410s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 6 of 10 (1761313696) service : cur 31 worst 31 (at 1761313285, 411s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 7 of 10 (1761313697) service : cur 31 worst 31 (at 1761313285, 412s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 8 of 10 (1761313698) service : cur 31 worst 31 (at 1761313285, 413s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 9 of 10 (1761313699) service : cur 31 worst 31 (at 1761313285, 414s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 10 of 10 (1761313700) service : cur 31 worst 31 (at 1761313285, 416s ago) 6 30 30 31 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress PASS 44b (20s) == replay-single test 44c: race in target handle connect ========================================================== 09:48:25 (1761313705) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre total: 100 create in 0.58 seconds: 173.29 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 1 seconds: 0.000000 unlinks/second Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (68s) == replay-single test 45: Handle failed close ============ 09:49:33 (1761313773) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 /mnt/lustre/f45.replay-single has type file OK PASS 45 (6s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 09:49:39 (1761313779) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (54s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:50:33 (1761313833) total: 20 open/close in 0.26 seconds: 76.84 ops/second Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.21 seconds: 97.06 ops/second - unlinked 0 (time 1761313929 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (100s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:52:13 (1761313933) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre total: 20 open/close in 0.19 seconds: 103.15 ops/second Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000216 total: 20 open/close in 0.25 seconds: 80.30 ops/second - unlinked 0 (time 1761314036 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (106s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 09:54:00 (1761314040) PASS 50 (10s) == replay-single test 52: time out lock replay (3764) ==== 09:54:11 (1761314051) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7735 fail_loc=0x80000157 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (84s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 09:55:35 (1761314135) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (53s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 09:56:28 (1761314188) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7735 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (61s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 09:57:29 (1761314249) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (52s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 09:58:21 (1761314301) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (40s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 09:59:01 (1761314341) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (52s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 09:59:53 (1761314393) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (46s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:00:40 (1761314440) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (41s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:01:20 (1761314480) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (50s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:02:10 (1761314530) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (50s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:03:00 (1761314580) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1936 1285752 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (61s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:04:01 (1761314641) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1936 1285752 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg335-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (62s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:05:03 (1761314703) fail_loc=0x8000012c - open/close 1008 (time 1761314717.43 total 10.01 last 100.73) - open/close 1958 (time 1761314727.44 total 20.01 last 94.93) total: 2500 open/close in 25.88 seconds: 96.60 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1761314817 ; total 0 ; last 0) total: 2500 unlinks in 17 seconds: 147.058823 unlinks/second PASS 58a (136s) == replay-single test 58b: test replay of setxattr op ==== 10:07:19 (1761314839) Starting client: oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) oleg335-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (56s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:08:15 (1761314895) Starting client: oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) PASS 58c (102s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:09:58 (1761314998) total: 200 open/close in 1.93 seconds: 103.74 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2072 1285616 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1761315007 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761315051 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (57s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:10:55 (1761315055) total: 800 open/close in 8.37 seconds: 95.62 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2148 1285540 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1456 3605564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2852 7211188 1% /mnt/lustre - unlinked 0 (time 1761315075 ; total 0 ; last 0) total: 800 unlinks in 5 seconds: 160.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (135s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:13:10 (1761315190) fail_loc=0x8000013a Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109811 s, 373 kB/s PASS 61b (77s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:14:27 (1761315267) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (48s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:15:15 (1761315315) Stopping /mnt/lustre-mds1 (opts:) on oleg335-server fail_loc=0x80000605 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg335-client: oleg335-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (29s) == replay-single test 62: don't mis-drop resent replay === 10:15:44 (1761315344) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1460 3605560 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1396 3605624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2856 7211184 1% /mnt/lustre total: 25 open/close in 0.30 seconds: 83.96 ops/second fail_loc=0x80000707 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1761315436 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (96s) == replay-single test 65a: AT: verify early replies ====== 10:17:20 (1761315440) at_history=8 at_history=8 debug=other fail_val=6000 fail_loc=0x8000050a replay-single test_65a: @@@@@@ FAIL: No early reply Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:1842:test_65a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6713:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:1849:main() Dumping lctl log to /tmp/testlogs//replay-single.test_65a.*.1761315470.log rsync: chown "/tmp/testlogs/.replay-single.test_65a.debug_log.oleg335-server.1761315470.log.WhICW4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_65a.dmesg.oleg335-server.1761315470.log.E43YLZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 65a (32s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:17:53 (1761315473) at_history=8 at_history=8 debug=other trace fail_val=6 fail_loc=0x224 fail_loc=0 00000100:00001000:0.0:1761315502.879118:0:2263:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@000000008c10db53 x1846867833417984/t0(0) o4->lustre-OST0000-osc-ffff9412100f7800@192.168.203.135@tcp:6/4 lens 4584/448 e 1 to 0 dl 1761315533 ref 2 fl Rpc:Qr/0/ffffffff rc 0/-1 job:'multiop.0' debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck portal 28 : cur 1 worst 31 (at 1761311347, 4162s ago) 1 30 31 31 portal 7 : cur 1 worst 1 (at 1761310814, 4695s ago) 1 1 1 1 portal 17 : cur 1 worst 1 (at 1761310924, 4585s ago) 1 0 0 1 portal 6 : cur 31 worst 31 (at 1761315507, 2s ago) 31 0 1 0 PASS 65b (39s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:18:31 (1761315511) at_history=8 at_history=8 portal 12 : cur 1 worst 31 (at 1761313285, 2251s ago) 1 28 28 28 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 31 worst 31 (at 1761313285, 2258s ago) 31 28 28 28 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 10 worst 31 (at 1761313285, 2270s ago) 31 28 28 28 fail_loc=0 portal 12 : cur 1 worst 31 (at 1761313285, 2280s ago) 31 28 28 28 Current MDT timeout 1, worst 31 PASS 66a (59s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:19:30 (1761315570) at_history=8 at_history=8 fail_val=6 fail_loc=0x50c fail_loc=0 network timeout orig 1, cur 6, worst 6 PASS 66b (61s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:20:31 (1761315631) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (59s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:21:30 (1761315690) at_history=8 at_history=8 Creating to objid 5537 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.23 seconds: 78.73 ops/second Connected clients: oleg335-client.virtnet oleg335-client.virtnet service : cur 1 worst 31 (at 1761315468, 253s ago) 1 1 0 1 phase 2 1 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg335-client.virtnet oleg335-client.virtnet service : cur 1 worst 31 (at 1761315468, 257s ago) 1 1 1 0 0 osc reconnect attempts on 2nd slow PASS 67b (38s) == replay-single test 68: AT: verify slowing locks ======= 10:22:08 (1761315728) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (80s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 10:23:28 (1761315808) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (2s) == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 10:23:30 (1761315810) Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg335-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=138495 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2232 1285456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25976 3552204 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7157760 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-client.virtnet: looking for dbench program oleg335-client.virtnet: /usr/bin/dbench oleg335-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg335-client.virtnet oleg335-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg335-client.virtnet' oleg335-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg335-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg335-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg335-client.virtnet at Fri Oct 24 10:23:33 EDT 2025 oleg335-client.virtnet: waiting for dbench pid 138527 oleg335-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg335-client.virtnet: oleg335-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg335-client.virtnet: failed to create barrier semaphore oleg335-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg335-client.virtnet: releasing clients oleg335-client.virtnet: 1 137 6.01 MB/sec warmup 1 sec latency 69.985 ms oleg335-client.virtnet: 1 207 4.04 MB/sec warmup 2 sec latency 85.981 ms oleg335-client.virtnet: 1 305 3.73 MB/sec warmup 3 sec latency 130.102 ms oleg335-client.virtnet: 1 397 3.51 MB/sec warmup 4 sec latency 56.612 ms oleg335-client.virtnet: 1 485 3.38 MB/sec warmup 5 sec latency 85.020 ms oleg335-client.virtnet: 1 641 3.50 MB/sec warmup 6 sec latency 72.062 ms oleg335-client.virtnet: 1 726 3.04 MB/sec warmup 7 sec latency 66.592 ms oleg335-client.virtnet: 1 799 2.68 MB/sec warmup 8 sec latency 69.472 ms oleg335-client.virtnet: 1 896 2.41 MB/sec warmup 9 sec latency 110.357 ms oleg335-client.virtnet: 1 980 2.19 MB/sec warmup 10 sec latency 85.482 ms oleg335-client.virtnet: 1 1147 2.09 MB/sec warmup 11 sec latency 51.983 ms oleg335-client.virtnet: 1 1221 1.92 MB/sec warmup 12 sec latency 57.685 ms oleg335-client.virtnet: 1 1403 2.00 MB/sec warmup 13 sec latency 60.168 ms oleg335-client.virtnet: 1 1499 1.86 MB/sec warmup 14 sec latency 66.647 ms oleg335-client.virtnet: 1 1609 1.74 MB/sec warmup 15 sec latency 64.386 ms oleg335-client.virtnet: 1 1695 1.64 MB/sec warmup 16 sec latency 62.245 ms oleg335-client.virtnet: 1 1855 1.55 MB/sec warmup 17 sec latency 37.085 ms oleg335-client.virtnet: 1 2018 1.47 MB/sec warmup 18 sec latency 45.020 ms oleg335-client.virtnet: 1 2191 1.41 MB/sec warmup 19 sec latency 36.305 ms oleg335-client.virtnet: 1 2386 1.40 MB/sec warmup 20 sec latency 46.145 ms oleg335-client.virtnet: 1 2554 1.37 MB/sec warmup 21 sec latency 39.898 ms oleg335-client.virtnet: 1 2890 1.48 MB/sec warmup 22 sec latency 33.092 ms oleg335-client.virtnet: 1 3177 1.48 MB/sec warmup 23 sec latency 35.641 ms oleg335-client.virtnet: 1 3688 0.40 MB/sec execute 1 sec latency 45.359 ms oleg335-client.virtnet: 1 3816 0.71 MB/sec execute 2 sec latency 104.919 ms oleg335-client.virtnet: 1 3910 0.57 MB/sec execute 3 sec latency 57.169 ms oleg335-client.virtnet: 1 3995 0.43 MB/sec execute 4 sec latency 50.873 ms oleg335-client.virtnet: 1 4075 0.35 MB/sec execute 5 sec latency 98.404 ms oleg335-client.virtnet: 1 4220 0.31 MB/sec execute 6 sec latency 64.530 ms oleg335-client.virtnet: 1 4304 0.30 MB/sec execute 7 sec latency 60.743 ms oleg335-client.virtnet: 1 4379 0.28 MB/sec execute 8 sec latency 68.432 ms oleg335-client.virtnet: 1 4484 0.28 MB/sec execute 9 sec latency 67.364 ms oleg335-client.virtnet: 1 4562 0.27 MB/sec execute 10 sec latency 56.052 ms oleg335-client.virtnet: 1 4721 0.34 MB/sec execute 11 sec latency 46.048 ms olegoleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2192 1285496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12120 3593984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37244 3569244 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49364 7163228 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 335-client.virtnet: 1 4817 0.32 MB/sec execute 12 sec latency 52.782 ms oleg335-client.virtnet: 1 5047 0.52 MB/sec execute 13 sec latency 38.606 ms oleg335-client.virtnet: 1 5162 0.49 MB/sec execute 14 sec latency 51.314 ms oleg335-client.virtnet: 1 5290 0.47 MB/sec execute 15 sec latency 40.530 ms oleg335-client.virtnet: 1 5413 0.45 MB/sec execute 16 sec latency 53.394 ms oleg335-client.virtnet: 1 5565 0.43 MB/sec execute 17 sec latency 61.201 ms oleg335-client.virtnet: 1 5732 0.42 MB/sec execute 18 sec latency 40.560 ms oleg335-client.virtnet: 1 5916 0.46 MB/sec execute 19 sec latency 81.686 ms oleg335-client.virtnet: 1 5988 0.44 MB/sec execute 20 sec latency 65.657 ms oleg335-client.virtnet: 1 6066 0.45 MB/sec execute 21 sec latency 80.246 ms oleg335-client.virtnet: 1 6315 0.58 MB/sec execute 22 sec latency 50.581 ms oleg335-client.virtnet: 1 6479 0.62 MB/sec execute 23 sec latency 65.026 ms oleg335-client.virtnet: 1 6727 0.62 MB/sec execute 24 sec latency 66.889 ms oleg335-client.virtnet: 1 6927 0.65 MB/sec execute 25 sec latency 49.660 ms oleg335-client.virtnet: 1 7122 0.74 MB/sec execute 26 sec latency 52.963 ms oleg335-client.virtnet: 1 7240 0.72 MB/sec execute 27 sec latency 48.294 ms oleg335-client.virtnet: 1 7348 0.73 MB/sec execute 28 sec latency 81.310 ms oleg335-client.virtnet: 1 7446 0.72 MB/sec execute 29 sec latency 53.889 ms oleg335-client.virtnet: 1 7523 0.69 MB/sec execute 30 sec latency 75.103 ms oleg335-client.virtnet: 1 7593 0.67 MB/sec execute 31 sec latency 72.293 ms oleg335-client.virtnet: 1 7766 0.65 MB/sec execute 32 sec latency 73.985 ms oleg335-client.virtnet: 1 7846 0.64 MB/sec execute 33 sec latency 93.080 ms oleg335-client.virtnet: 1 7963 0.63 MB/sec execute 34 sec latency 55.052 ms oleg335-client.virtnet: 1 8036 0.62 MB/sec execute 35 sec latency 52.725 ms oleg335-client.virtnet: 1 8147 0.63 MB/sec execute 36 sec latency 60.744 ms oleg335-client.virtnet: 1 8245 0.62 MB/sec execute 37 sec latency 67.018 ms oleg335-client.virtnet: 1 8350 0.60 MB/sec execute 38 sec latency 59.266 ms oleg335-client.virtnet: 1 8519 0.66 MB/sec execute 39 sec latency 62.460 ms oleg335-client.virtnet: 1 8608 0.65 MB/sec execute 40 sec latency 45.263 ms oleg335-client.virtnet: 1 8729 0.63 MB/sec execute 41 sec latency 55.124 ms oleg335-client.virtnet: 1 8799 0.62 MB/sec execute 42 sec latency 74.225 ms oleg335-client.virtnet: 1 8878 0.61 MB/sec execute 43 sec latency 67.719 ms oleg335-client.virtnet: 1 8945 0.60 MB/sec execute 44 sec latency 56.093 ms oleg335-client.virtnet: 1 8986 0.58 MB/sec execute 45 sec latency 626.289 ms oleg335-client.virtnet: 1 8986 0.57 MB/sec execute 46 sec latency 1627.266 ms oleg335-client.virtnet: 1 8986 0.56 MB/sec execute 47 sec latency 2628.836 ms oleg335-client.virtnet: 1 8986 0.55 MB/sec execute 48 sec latency 3629.220 ms oleg335-client.virtnet: 1 8986 0.54 MB/sec execute 49 sec latency 4633.062 ms oleg335-client.virtnet: 1 8986 0.52 MB/sec execute 50 sec latency 5634.121 ms oleg335-client.virtnet: 1 8986 0.51 MB/sec execute 51 sec latency 6636.728 ms oleg335-client.virtnet: 1 8986 0.50 MB/sec execute 52 sec latency 7636.915 ms oleg335-client.virtnet: 1 8986 0.49 MB/sec execute 53 sec latency 8640.356 ms oleg335-client.virtnet: 1 8986 0.49 MB/sec execute 54 sec latency 9644.455 ms oleg335-client.virtnet: 1 8986 0.48 MB/sec execute 55 sec latency 10645.347 ms oleg335-client.virtnet: 1 8986 0.47 MB/sec execute 56 sec lFailover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12020 3592144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37364 3568984 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49384 7161128 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 atency 11648.754 ms oleg335-client.virtnet: 1 8986 0.46 MB/sec execute 57 sec latency 12651.739 ms oleg335-client.virtnet: 1 8986 0.45 MB/sec execute 58 sec latency 13651.936 ms oleg335-client.virtnet: 1 8986 0.44 MB/sec execute 59 sec latency 14652.453 ms oleg335-client.virtnet: 1 8986 0.44 MB/sec execute 60 sec latency 15652.631 ms oleg335-client.virtnet: 1 8986 0.43 MB/sec execute 61 sec latency 16652.759 ms oleg335-client.virtnet: 1 8986 0.42 MB/sec execute 62 sec latency 17655.196 ms oleg335-client.virtnet: 1 8986 0.42 MB/sec execute 63 sec latency 18655.661 ms oleg335-client.virtnet: 1 8986 0.41 MB/sec execute 64 sec latency 19656.782 ms oleg335-client.virtnet: 1 8986 0.40 MB/sec execute 65 sec latency 20657.076 ms oleg335-client.virtnet: 1 8986 0.40 MB/sec execute 66 sec latency 21657.293 ms oleg335-client.virtnet: 1 8986 0.39 MB/sec execute 67 sec latency 22657.480 ms oleg335-client.virtnet: 1 8986 0.39 MB/sec execute 68 sec latency 23658.147 ms oleg335-client.virtnet: 1 8986 0.38 MB/sec execute 69 sec latency 24659.253 ms oleg335-client.virtnet: 1 8986 0.37 MB/sec execute 70 sec latency 25666.019 ms oleg335-client.virtnet: 1 8986 0.37 MB/sec execute 71 sec latency 26666.907 ms oleg335-client.virtnet: 1 8997 0.36 MB/sec execute 72 sec latency 27508.066 ms oleg335-client.virtnet: 1 9105 0.36 MB/sec execute 73 sec latency 50.645 ms oleg335-client.virtnet: 1 9203 0.36 MB/sec execute 74 sec latency 67.797 ms oleg335-client.virtnet: 1 9315 0.35 MB/sec execute 75 sec latency 54.851 ms oleg335-client.virtnet: 1 9471 0.37 MB/sec execute 76 sec latency 72.996 ms oleg335-client.virtnet: 1 9535 0.36 MB/sec execute 77 sec latency 99.055 ms oleg335-client.virtnet: 1 9670 0.37 MB/sec execute 78 sec latency 72.184 ms oleg335-client.virtnet: 1 9991 0.41 MB/sec execute 79 sec latency 37.036 ms oleg335-client.virtnet: 1 10123 0.42 MB/sec execute 80 sec latency 65.781 ms oleg335-client.virtnet: 1 10459 0.44 MB/sec execute 81 sec latency 43.607 ms oleg335-client.virtnet: 1 10660 0.47 MB/sec execute 82 sec latency 49.253 ms oleg335-client.virtnet: 1 10773 0.46 MB/sec execute 83 sec latency 40.577 ms oleg335-client.virtnet: 1 10889 0.47 MB/sec execute 84 sec latency 66.649 ms oleg335-client.virtnet: 1 10975 0.47 MB/sec execute 85 sec latency 66.420 ms oleg335-client.virtnet: 1 11069 0.46 MB/sec execute 86 sec latency 78.768 ms oleg335-client.virtnet: 1 11166 0.46 MB/sec execute 87 sec latency 58.043 ms oleg335-client.virtnet: 1 11300 0.45 MB/sec execute 88 sec latency 59.080 ms oleg335-client.virtnet: 1 11381 0.45 MB/sec execute 89 sec latency 84.492 ms oleg335-client.virtnet: 1 11449 0.45 MB/sec execute 90 sec latency 69.604 ms oleg335-client.virtnet: 1 11549 0.45 MB/sec execute 91 sec latency 50.157 ms oleg335-client.virtnet: 1 11648 0.44 MB/sec execute 92 sec latency 52.236 ms oleg335-client.virtnet: 1 11812 0.45 MB/sec execute 93 sec latency 53.701 ms oleg335-client.virtnet: 1 11974 0.46 MB/sec execute 94 sec latency 39.420 ms oleg335-client.virtnet: 1 12156 0.47 MB/sec execute 95 sec latency 38.992 ms oleg335-client.virtnet: 1 12301 0.47 MB/sec execute 96 sec latency 51.624 ms oleg335-client.virtnet: 1 12436 0.47 MB/sec execute 97 sec latency 47.181 ms oleg335-client.virtnet: 1 12604 0.46 MB/sec execute 98 sec latency 38.732 ms oleg335-client.virtnet: 1 12763 0.46 MB/sec execute 99 sec latency 49.488 ms oleg335-client.virtnet: 1 12996 0.47 MB/sec execute 100 sec latency 32.026 ms oleg335-client.virtnet: Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 1 13099 0.47 MB/sec execute 101 sec latency 43.857 ms oleg335-client.virtnet: 1 13236 0.47 MB/sec execute 102 sec latency 53.060 ms oleg335-client.virtnet: 1 13526 0.50 MB/sec execute 103 sec latency 60.933 ms oleg335-client.virtnet: 1 13646 0.50 MB/sec execute 104 sec latency 53.549 ms oleg335-client.virtnet: 1 13920 0.52 MB/sec execute 105 sec latency 58.401 ms oleg335-client.virtnet: 1 14081 0.52 MB/sec execute 106 sec latency 53.335 ms oleg335-client.virtnet: 1 14234 0.54 MB/sec execute 107 sec latency 74.142 ms oleg335-client.virtnet: 1 14355 0.53 MB/sec execute 108 sec latency 58.097 ms oleg335-client.virtnet: 1 14484 0.54 MB/sec execute 109 sec latency 59.955 ms oleg335-client.virtnet: 1 14598 0.54 MB/sec execute 110 sec latency 62.710 ms oleg335-client.virtnet: 1 14722 0.53 MB/sec execute 111 sec latency 35.020 ms oleg335-client.virtnet: 1 14881 0.53 MB/sec execute 112 sec latency 60.439 ms oleg335-client.virtnet: 1 14993 0.53 MB/sec execute 113 sec latency 48.230 ms oleg335-client.virtnet: 1 15119 0.52 MB/sec execute 114 sec latency 44.757 ms oleg335-client.virtnet: 1 15246 0.53 MB/sec execute 115 sec latency 41.290 ms oleg335-client.virtnet: 1 15381 0.53 MB/sec execute 116 sec latency 45.120 ms oleg335-client.virtnet: 1 15479 0.52 MB/sec execute 117 sec latency 56.255 ms oleg335-client.virtnet: 1 15657 0.54 MB/sec execute 118 sec latency 51.325 ms oleg335-client.virtnet: 1 15775 0.54 MB/sec execute 119 sec latency 46.493 ms oleg335-client.virtnet: 1 cleanup 120 sec oleg335-client.virtnet: 1 cleanup 121 sec oleg335-client.virtnet: 1 cleanup 122 sec oleg335-client.virtnet: 0 cleanup 123 sec oleg335-client.virtnet: oleg335-client.virtnet: Operation Count AvgLat MaxLat oleg335-client.virtnet: ---------------------------------------- oleg335-client.virtnet: NTCreateX 2211 35.992 27508.046 oleg335-client.virtnet: Close 1597 5.473 34.439 oleg335-client.virtnet: Rename 92 32.669 99.033 oleg335-client.virtnet: Unlink 464 8.622 57.755 oleg335-client.virtnet: Qpathinfo 1972 5.306 81.659 oleg335-client.virtnet: Qfileinfo 327 0.328 9.536 oleg335-client.virtnet: Qfsinfo 362 0.418 28.574 oleg335-client.virtnet: Sfileinfo 184 17.710 52.895 oleg335-client.virtnet: Find 736 1.761 73.964 oleg335-client.virtnet: WriteX 1055 4.317 47.010 oleg335-client.virtnet: ReadX 3211 0.031 10.848 oleg335-client.virtnet: LockX 6 5.755 7.188 oleg335-client.virtnet: UnlockX 6 4.160 6.091 oleg335-client.virtnet: Flush 156 29.692 93.065 oleg335-client.virtnet: oleg335-client.virtnet: Throughput 0.539391 MB/sec 1 clients 1 procs max_latency=27508.066 ms oleg335-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg335-client.virtnet at Fri Oct 24 10:26:00 EDT 2025 with return code 0 oleg335-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg335-client.virtnet oleg335-client.virtnet: /mnt/lustre/d70b.replay-single/oleg335-client.virtnet /mnt/lustre/d70b.replay-single/oleg335-client.virtnet oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg335-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg335-client.virtnet: removed directory 'clients/client0' oleg335-client.virtnet: removed directory 'clients' oleg335-client.virtnet: removed 'client.txt' oleg335-client.virtnet: /mnt/lustre/d70b.replay-single/oleg335-client.virtnet oleg335-client.virtnet: dbench successfully finished oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70b (161s) == replay-single test 70c: tar 2mdts recovery ============ 10:26:11 (1761315971) Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar 142654 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4772 1282916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4424 1283264 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 20948 3580480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 3076 3599488 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 24024 7179968 1% /mnt/lustre tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets test_70c fail mds2 1 times Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4744 1282944 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4412 1283276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 3868 3598952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19196 3582112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23064 7181064 1% /mnt/lustre test_70c fail mds1 2 times Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (404s) == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 10:32:55 (1761316375) Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started 145674 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4252 1283436 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3880 1283808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7210584 1% /mnt/lustre test_70d fail mds1 1 times Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4298: 145674 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (187s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:36:02 (1761316562) debug=+ha Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started PID=150647 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3716 1283972 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3172 1284516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7210584 1% /mnt/lustre test_70e fail mds2 1 times Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7210584 1% /mnt/lustre test_70e fail mds1 2 times Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (359s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 10:42:01 (1761316921) mount clients oleg335-client.virtnet ... Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1285904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6584 3583812 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6520 3583876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 13104 7167688 1% /mnt/lustre ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Started lustre-OST0000 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1285904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6584 3584860 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3601500 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 7984 7186360 1% /mnt/lustre ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9412100f7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9412100f7800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg335-client.virtnet' ... PASS 70f (110s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 10:43:51 (1761317031) Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started 171808 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3196 1284492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3152 1284536 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3292 1284396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3256 1284432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail mds2 mds1 1 times Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4298: 171808 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (339s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 10:49:30 (1761317370) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2740 1284948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (56s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 10:50:26 (1761317426) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7735 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2740 1284948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (56s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 10:51:22 (1761317482) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:) Stopping client oleg335-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg335-server Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (57s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 10:52:19 (1761317539) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2744 1284944 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.23 seconds: 86.59 ops/second PASS 80a (54s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 10:53:13 (1761317593) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.31 seconds: 65.14 ops/second PASS 80b (57s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 10:54:10 (1761317650) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2820 1284868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2596 1285092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2820 1284868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2596 1285092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.26 seconds: 77.99 ops/second PASS 80c (95s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 10:55:45 (1761317745) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2856 1284832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.25 seconds: 81.13 ops/second PASS 80d (97s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 10:57:23 (1761317843) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.28 seconds: 71.74 ops/second PASS 80e (55s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 10:58:17 (1761317897) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.22 seconds: 90.98 ops/second PASS 80f (42s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 10:59:00 (1761317940) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.34 seconds: 59.01 ops/second PASS 80g (98s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 11:00:38 (1761318038) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.37 seconds: 53.51 ops/second PASS 80h (94s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 11:02:12 (1761318132) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (45s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 11:02:57 (1761318177) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2816 1284872 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (50s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 11:03:47 (1761318227) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (94s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 11:05:21 (1761318321) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2816 1284872 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2816 1284872 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (97s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 11:06:58 (1761318418) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (55s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 11:07:54 (1761318474) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (51s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 11:08:45 (1761318525) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (106s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 11:10:32 (1761318632) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1464 3605556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2864 7211176 1% /mnt/lustre Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Failing mds2 on oleg335-server Stopping /mnt/lustre-mds2 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg335-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (98s) == replay-single test 84a: stale open during export disconnect ========================================================== 11:12:09 (1761318729) fail_loc=0x80000144 total: 1 open/close in 0.04 seconds: 25.86 ops/second oleg335-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg335-client: oleg335-client: ssh exited with exit code 5 PASS 84a (9s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 11:12:18 (1761318738) before recovery: unused locks count = 301 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (41s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 11:12:59 (1761318779) before recovery: unused locks count = 100 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (50s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 11:13:49 (1761318829) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:) Stopping client oleg335-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 86 (23s) == replay-single test 87a: write replay ================== 11:14:12 (1761318852) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1664 3605356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1600 3605420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3264 7210776 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.231932 s, 36.2 MB/s Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.167914 s, 50.0 MB/s PASS 87a (43s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 11:14:55 (1761318895) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9856 3597164 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1600 3605420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11456 7202584 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.334111 s, 25.1 MB/s 8+0 records in 8+0 records out 8 bytes copied, 0.00805437 s, 1.0 kB/s Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes copied, 0.00918442 s, 7.8 kB/s PASS 87b (42s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 11:15:37 (1761318937) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1600 3605420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11460 7202580 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2784 1284904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1285120 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1600 3605420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11460 7202580 1% /mnt/lustre before test: last_id = 8993, next_id = 8962 Creating to objid 8993 on ost lustre-OST0000... total: 33 open/close in 0.43 seconds: 76.62 ops/second total: 8 open/close in 0.07 seconds: 108.22 ops/second before recovery: last_id = 9025, next_id = 9002 Stopping /mnt/lustre-mds1 (opts:) on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server Failover mds1 to oleg335-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg335-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 9033, next_id = 9002 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0984525 s, 5.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0423266 s, 12.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.077451 s, 6.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0599668 s, 8.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0505705 s, 10.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0464301 s, 11.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.086084 s, 6.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0449055 s, 11.7 MB/s -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8962 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8963 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8964 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8965 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8966 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8967 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8968 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8969 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8970 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8971 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8972 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8973 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8974 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8975 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8976 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8977 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8978 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8979 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8980 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8981 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8982 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8983 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8984 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8985 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8986 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8987 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8988 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8989 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8990 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8991 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8992 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8993 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8994 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8995 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8996 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8997 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8998 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-8999 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-9000 -rw-r--r-- 1 root root 0 Oct 24 11:15 /mnt/lustre/d88.replay-single/f-9001 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9005 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9006 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9007 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9008 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9009 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9010 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9011 -rw-r--r-- 1 root root 524288 Oct 24 11:16 /mnt/lustre/d88.replay-single/f-9012 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0530879 s, 9.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0411559 s, 12.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0402424 s, 13.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0426056 s, 12.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0404448 s, 13.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0709415 s, 7.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0356255 s, 14.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0710497 s, 7.4 MB/s PASS 88 (84s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 11:17:01 (1761319021) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. Waiting for MDT destroys to complete 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.153953 s, 68.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg335-server Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg335-client.virtnet: -o user_xattr,flock oleg335-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7646580 free_after: 7646580 PASS 89 (132s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 11:19:14 (1761319154) Waiting 90s for '2' Waiting 70s for '2' Waiting 60s for '2' Waiting 50s for '2' Waiting 40s for '2' Waiting 30s for '2' Waiting 20s for '2' Waiting 10s for '2' Waiting 0s for '2' Update not seen after 90s: want '2' got '1' replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:3525:wait_osts_up() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3350:test_90() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6713:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3425:main() Dumping lctl log to /tmp/testlogs//replay-single.test_90.*.1761319247.log rsync: chown "/tmp/testlogs/.replay-single.test_90.debug_log.oleg335-server.1761319247.log.Cs6Si0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_90.dmesg.oleg335-server.1761319247.log.n9cO9q" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 90 (95s) == replay-single test 93a: replay + reconnect ============ 11:20:49 (1761319249) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00280849 s, 365 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg335-server Stopping /mnt/lustre-ost1 (opts:) on oleg335-server reboot facets: ost1 Failover ost1 to oleg335-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (67s) == replay-single test 93b: replay + reconnect on mds ===== 11:21:56 (1761319316) total: 20 open/close in 0.11 seconds: 184.66 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server reboot facets: mds1 Failover mds1 to oleg335-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (119s) == replay-single test complete, duration 8919 sec ======== 11:23:55 (1761319435) replay-single: FAIL: test_65a No early reply replay-single: FAIL: test_90 wait_update OSTs up on MDT0000 failed