-----============= acceptance-small: replay-single ============----- Tue Apr 14 17:10:45 EDT 2026 oleg260-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 oleg260-client.virtnet: executing check_config_client /mnt/lustre oleg260-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg260-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa09d0673d000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa09d0673d000.idle_timeout=debug disable quota as required oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 == replay-single test 0a: empty replay =================== 17:11:30 (1776201090) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (68s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 17:12:39 (1776201159) Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.35 seconds: 57.72 ops/second - unlinked 0 (time 1776201196 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (42s) == replay-single test 0c: check replay-barrier =========== 17:13:21 (1776201201) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (109s) == replay-single test 0d: expired recovery with no clients ========================================================== 17:15:10 (1776201310) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre PASS 0d (107s) == replay-single test 1: simple create =================== 17:16:57 (1776201417) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (51s) == replay-single test 2a: touch ========================== 17:17:48 (1776201468) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (59s) == replay-single test 2b: touch ========================== 17:18:48 (1776201528) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (69s) == replay-single test 2c: setstripe replay =============== 17:19:57 (1776201597) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (54s) == replay-single test 2d: setdirstripe replay ============ 17:20:52 (1776201652) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (57s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 17:21:49 (1776201709) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (59s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 17:22:48 (1776201768) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (56s) == replay-single test 3b: replay failed open -ENOMEM ===== 17:23:44 (1776201824) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (69s) == replay-single test 3c: replay failed open -ENOMEM ===== 17:24:53 (1776201893) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (64s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 17:25:57 (1776201957) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (71s) == replay-single test 4b: |x| rm 10 files ================ 17:27:08 (1776202028) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (66s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 17:28:14 (1776202094) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7518208 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (81s) == replay-single test 6a: mkdir + contained create ======= 17:29:35 (1776202175) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (55s) == replay-single test 6b: |X| rmdir ====================== 17:30:30 (1776202230) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (64s) == replay-single test 7: mkdir |X| contained create ====== 17:31:34 (1776202294) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (65s) == replay-single test 8: creat open |X| close ============ 17:32:40 (1776202360) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (62s) == replay-single test 9: |X| create (same inum/gen) ====== 17:33:42 (1776202422) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (60s) == replay-single test 10: create |X| rename unlink ======= 17:34:43 (1776202483) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (49s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 17:35:32 (1776202532) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre new old Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (61s) == replay-single test 12: open, unlink |X| close ========= 17:36:33 (1776202593) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (66s) == replay-single test 13: open chmod 0 |x| write close === 17:37:39 (1776202659) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (64s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 17:38:43 (1776202723) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (65s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 17:39:48 (1776202788) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (62s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 17:40:51 (1776202851) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (68s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 17:41:58 (1776202918) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (60s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 17:42:58 (1776202978) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 pid: 50734 will close Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (55s) == replay-single test 19: mcreate, open, write, rename === 17:43:53 (1776203033) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre old Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (63s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:44:56 (1776203096) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (64s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 17:46:00 (1776203160) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown oleg260-client: error: invalid path '/mnt/lustre': Input/output error 5769+0 records in 5768+0 records out 23625728 bytes (24 MB, 23 MiB) copied, 3.32319 s, 7.1 MB/s pdsh@oleg260-client: oleg260-client: ssh exited with exit code 5 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg260-server: *.lustre-MDT0000.recovery_status status: COMPLETE sleep 5 for ZFS MDS Waiting for MDT destroys to complete before 6144, after 6144 PASS 20b (84s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 17:47:24 (1776203244) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 -rw-r--r-- 1 root root 1 Apr 14 17:47 /mnt/lustre/f20c.replay-single oleg260-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg260-client: oleg260-client: ssh exited with exit code 5 PASS 20c (10s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:47:35 (1776203255) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (63s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:48:39 (1776203319) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (63s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:49:42 (1776203382) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (56s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 17:50:38 (1776203438) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (56s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:51:34 (1776203494) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (51s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:52:25 (1776203545) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (47s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:53:13 (1776203593) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (69s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:54:22 (1776203662) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (64s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:55:26 (1776203726) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (65s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:56:32 (1776203792) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (62s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 17:57:33 (1776203853) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (61s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 17:58:34 (1776203914) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 PASS 32 (8s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 17:58:43 (1776203923) total: 10 open/close in 0.17 seconds: 58.64 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.29 seconds: 34.96 ops/second PASS 33a (28s) == replay-single test 33b: test fid seq allocation ======= 17:59:11 (1776203951) fail_loc=0x1311 total: 10 open/close in 0.27 seconds: 37.42 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.25 seconds: 40.80 ops/second PASS 33b (32s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 17:59:43 (1776203983) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (41s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 18:00:24 (1776204024) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (40s) == replay-single test 36: don't resend cancel ============ 18:01:04 (1776204064) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 36 (55s) == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 18:02:00 (1776204120) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (32s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:02:32 (1776204152) total: 800 open/close in 9.47 seconds: 84.49 ops/second - unlinked 0 (time 1776204169 ; total 0 ; last 0) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776204228 ; total 1 ; last 1) total: 400 unlinks in 6 seconds: 66.666664 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (89s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:04:01 (1776204241) total: 800 open/close in 9.41 seconds: 85.02 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776204260 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776204307 ; total 0 ; last 0) total: 400 unlinks in 5 seconds: 80.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (79s) == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 18:05:20 (1776204320) SKIP: replay-single test_40 layout_lock needs MDS connection for IO SKIP 40 (3s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:05:23 (1776204323) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00734007 s, 558 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016896 s, 242 kB/s PASS 41 (9s) == replay-single test 42: recovery after ost failure ===== 18:05:32 (1776204332) total: 800 open/close in 9.72 seconds: 82.28 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776204357 ; total 0 ; last 0) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second debug=-1 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 wait for MDS to timeout and recover - unlinked 0 (time 1776204436 ; total 1 ; last 1) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (114s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:07:26 (1776204446) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (66s) == replay-single test 44a: race in target handle connect ========================================================== 18:08:32 (1776204512) at_max=40 1 of 10 (1776204517) service : cur 1 worst 1 (at 1776204477, 41s ago) 1 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 2 of 10 (1776204525) service : cur 6 worst 6 (at 1776204525, 1s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 3 of 10 (1776204532) service : cur 6 worst 6 (at 1776204525, 8s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 4 of 10 (1776204540) service : cur 6 worst 6 (at 1776204525, 16s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 5 of 10 (1776204547) service : cur 6 worst 6 (at 1776204525, 23s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 6 of 10 (1776204554) service : cur 6 worst 6 (at 1776204525, 31s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 7 of 10 (1776204562) service : cur 6 worst 6 (at 1776204525, 38s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 8 of 10 (1776204568) service : cur 6 worst 6 (at 1776204525, 44s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 9 of 10 (1776204576) service : cur 6 worst 6 (at 1776204525, 52s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 10 of 10 (1776204583) service : cur 6 worst 6 (at 1776204525, 59s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (85s) == replay-single test 44b: race in target handle connect ========================================================== 18:09:58 (1776204598) 1 of 10 (1776204599) service : cur 6 worst 6 (at 1776204525, 75s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 2 of 10 (1776204607) service : cur 6 worst 6 (at 1776204525, 83s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 3 of 10 (1776204609) service : cur 6 worst 6 (at 1776204525, 84s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 4 of 10 (1776204610) service : cur 6 worst 6 (at 1776204525, 86s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 5 of 10 (1776204612) service : cur 6 worst 6 (at 1776204525, 88s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 6 of 10 (1776204614) service : cur 6 worst 6 (at 1776204525, 89s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 7 of 10 (1776204615) service : cur 6 worst 6 (at 1776204525, 91s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 8 of 10 (1776204617) service : cur 6 worst 6 (at 1776204525, 92s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 9 of 10 (1776204618) service : cur 6 worst 6 (at 1776204525, 94s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 10 of 10 (1776204620) service : cur 6 worst 6 (at 1776204525, 96s ago) 6 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress PASS 44b (29s) == replay-single test 44c: race in target handle connect ========================================================== 18:10:26 (1776204626) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 100 create in 0.71 seconds: 141.19 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (80s) == replay-single test 45: Handle failed close ============ 18:11:46 (1776204706) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 /mnt/lustre/f45.replay-single has type file OK PASS 45 (7s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:11:53 (1776204713) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (68s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:13:01 (1776204781) total: 20 open/close in 0.39 seconds: 50.70 ops/second Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.28 seconds: 72.69 ops/second - unlinked 0 (time 1776204887 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (110s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:14:51 (1776204891) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 20 open/close in 0.27 seconds: 72.98 ops/second Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000216 total: 20 open/close in 0.31 seconds: 65.12 ops/second - unlinked 0 (time 1776205001 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (115s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:16:46 (1776205006) PASS 50 (14s) == replay-single test 52: time out lock replay (3764) ==== 18:17:00 (1776205020) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7147 fail_loc=0x80000157 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (68s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:18:08 (1776205088) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (54s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:19:03 (1776205143) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7147 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (61s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:20:03 (1776205203) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (52s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:20:55 (1776205255) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (60s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:21:56 (1776205316) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (66s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:23:01 (1776205381) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (52s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:23:53 (1776205433) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (60s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 18:24:53 (1776205493) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (51s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 18:25:44 (1776205544) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (16s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 18:26:00 (1776205560) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (66s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 18:27:06 (1776205626) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg260-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (67s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 18:28:13 (1776205693) fail_loc=0x8000012c - open/close 865 (time 1776205707.69 total 10.01 last 86.39) - open/close 1702 (time 1776205717.69 total 20.02 last 83.67) - open/close 2453 (time 1776205727.72 total 30.04 last 74.93) total: 2500 open/close in 30.90 seconds: 80.89 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4224 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1776205824 ; total 0 ; last 0) total: 2500 unlinks in 21 seconds: 119.047623 unlinks/second PASS 58a (159s) == replay-single test 58b: test replay of setxattr op ==== 18:30:52 (1776205852) Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) oleg260-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (62s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 18:31:54 (1776205914) Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) PASS 58c (105s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 18:33:41 (1776206021) total: 200 open/close in 2.99 seconds: 66.90 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776206032 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776206081 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second PASS 60 (67s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 18:34:48 (1776206088) - open/close 780 (time 1776206101.97 total 10.01 last 77.95) total: 800 open/close in 10.19 seconds: 78.50 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776206109 ; total 0 ; last 0) total: 800 unlinks in 5 seconds: 160.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (138s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 18:37:06 (1776206226) fail_loc=0x8000013a Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0156942 s, 261 kB/s PASS 61b (100s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 18:38:46 (1776206326) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (62s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 18:39:48 (1776206388) Stopping /mnt/lustre-mds1 (opts:) on oleg260-server fail_loc=0x80000605 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg260-client: oleg260-server: ssh exited with exit code 95 Start of lustre-mdt1/mdt1 on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (34s) == replay-single test 62: don't mis-drop resent replay === 18:40:22 (1776206422) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.46 seconds: 54.14 ops/second fail_loc=0x80000707 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1776206516 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (98s) == replay-single test 65a: AT: verify early replies ====== 18:42:00 (1776206520) at_history=8 at_history=8 debug=other fail_val=6000 fail_loc=0x8000050a 00000100:00001000:2.0:1776206553.373732:0:125896:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@000000001c01ed65 x1862481627020608/t0(0) o101->lustre-MDT0000-mdc-ffffa09d06973800@192.168.202.160@tcp:12/10 lens 664/66264 e 1 to 0 dl 1776206584 ref 2 fl Rpc:PQr/0/ffffffff rc 0/-1 job:'createmany.0' portal 12 : cur 6 worst 31 (at 1776203794, 2766s ago) 31 28 28 28 portal 23 : cur 1 worst 31 (at 1776204252, 2308s ago) 1 10 1 1 portal 30 : cur 1 worst 1 (at 1776201420, 5140s ago) 1 1 0 0 portal 17 : cur 1 worst 1 (at 1776201602, 4958s ago) 1 1 1 1 portal 12 : cur 6 worst 31 (at 1776203794, 2775s ago) 31 28 28 28 portal 23 : cur 1 worst 31 (at 1776204252, 2317s ago) 1 10 1 1 portal 30 : cur 1 worst 1 (at 1776201420, 5149s ago) 1 1 0 0 portal 17 : cur 1 worst 1 (at 1776201602, 4967s ago) 1 1 1 1 PASS 65a (54s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 18:42:54 (1776206574) at_history=8 at_history=8 debug=other trace fail_val=6 fail_loc=0x224 fail_loc=0 00000100:00001000:2.0:1776206607.004844:0:2222:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (26s) req@000000005f97726e x1862481627028800/t0(0) o4->lustre-OST0000-osc-ffffa09d06973800@192.168.202.160@tcp:6/4 lens 4584/448 e 1 to 0 dl 1776206638 ref 2 fl Rpc:Qr/0/ffffffff rc 0/-1 job:'multiop.0' portal 28 : cur 1 worst 31 (at 1776204179, 2435s ago) 31 31 31 31 portal 7 : cur 1 worst 1 (at 1776201411, 5203s ago) 1 1 1 1 portal 17 : cur 1 worst 1 (at 1776201527, 5087s ago) 1 0 0 0 portal 6 : cur 31 worst 31 (at 1776206612, 2s ago) 31 1 0 0 PASS 65b (45s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 18:43:39 (1776206619) at_history=8 at_history=8 portal 12 : cur 1 worst 31 (at 1776203794, 2856s ago) 31 28 28 28 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 32 worst 32 (at 1776206656, 4s ago) 32 32 28 28 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 10 worst 32 (at 1776206656, 17s ago) 32 32 28 28 fail_loc=0 portal 12 : cur 1 worst 32 (at 1776206656, 29s ago) 32 32 28 28 Current MDT timeout 1, worst 32 PASS 66a (73s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 18:44:52 (1776206692) at_history=8 at_history=8 fail_val=6 fail_loc=0x50c fail_loc=0 network timeout orig 1, cur 7, worst 7 PASS 66b (60s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 18:45:52 (1776206752) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (64s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 18:46:56 (1776206816) at_history=8 at_history=8 Creating to objid 5665 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 17 open/close in 0.27 seconds: 63.14 ops/second Connected clients: oleg260-client.virtnet oleg260-client.virtnet service : cur 1 worst 1 (at 1776200983, 5868s ago) 1 1 0 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg260-client.virtnet oleg260-client.virtnet service : cur 1 worst 1 (at 1776200983, 5873s ago) 1 0 1 1 0 osc reconnect attempts on 2nd slow PASS 67b (45s) == replay-single test 68: AT: verify slowing locks ======= 18:47:42 (1776206862) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (86s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 18:49:08 (1776206948) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (3s) == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 18:49:11 (1776206951) Starting client oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg260-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=131417 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 30720 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3755008 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7491584 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 oleg260-client.virtnet: looking for dbench program oleg260-client.virtnet: /usr/bin/dbench oleg260-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg260-client.virtnet oleg260-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg260-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg260-client.virtnet' oleg260-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg260-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg260-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg260-client.virtnet at Tue Apr 14 18:49:14 EDT 2026 oleg260-client.virtnet: waiting for dbench pid 131449 oleg260-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg260-client.virtnet: oleg260-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg260-client.virtnet: failed to create barrier semaphore oleg260-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg260-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg260-client.virtnet: releasing clients oleg260-client.virtnet: 1 71 3.13 MB/sec warmup 1 sec latency 137.111 ms oleg260-client.virtnet: 1 122 2.74 MB/sec warmup 2 sec latency 215.410 ms oleg260-client.virtnet: 1 157 2.18 MB/sec warmup 3 sec latency 180.069 ms oleg260-client.virtnet: 1 207 2.02 MB/sec warmup 4 sec latency 123.175 ms oleg260-client.virtnet: 1 289 2.14 MB/sec warmup 5 sec latency 120.724 ms oleg260-client.virtnet: 1 331 2.00 MB/sec warmup 6 sec latency 158.484 ms oleg260-client.virtnet: 1 366 1.86 MB/sec warmup 7 sec latency 125.968 ms oleg260-client.virtnet: 1 421 1.86 MB/sec warmup 8 sec latency 169.789 ms oleg260-client.virtnet: 1 477 1.85 MB/sec warmup 9 sec latency 179.643 ms oleg260-client.virtnet: 1 512 1.77 MB/sec warmup 10 sec latency 193.142 ms oleg260-client.virtnet: 1 608 1.90 MB/sec warmup 11 sec latency 132.948 ms oleg260-client.virtnet: 1 678 1.76 MB/sec warmup 12 sec latency 119.519 ms oleg260-client.virtnet: 1 721 1.64 MB/sec warmup 13 sec latency 99.383 ms oleg260-client.virtnet: 1 733 1.52 MB/sec warmup 14 sec latency 256.864 ms oleg260-client.virtnet: 1 785 1.43 MB/sec warmup 15 sec latency 166.589 ms oleg260-client.virtnet: 1 801 1.34 MB/sec warmup 16 sec latency 492.617 ms oleg260-client.virtnet: 1 811 1.26 MB/sec warmup 17 sec latency 815.566 ms oleg260-client.virtnet: 1 811 1.19 MB/sec warmup 18 sec latency 1822.022 ms oleg260-client.virtnet: 1 811 1.13 MB/sec warmup 19 sec latency 2822.349 ms oleg260-client.virtnet: 1 811 1.07 MB/sec warmup 20 sec latency 3822.772 ms oleg260-client.virtnet: 1 811 1.02 MB/sec warmup 21 sec latency 4824.377 ms oleg260-client.virtnet: 1 811 0.98 MB/sec warmup 22 sec latency 5825.097 ms oleg260-client.virtnet: 1 811 0.93 MB/sec warmup 23 sec latency 6828.770 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 1 sec latency 8829.939 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 2 sec latency 9830.141 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 3 sec latency 10831.946 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 4 sec latency 11832.166 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 5 sec latency 12834.020 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 6 sec latency 13834.270 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 7 sec latency 14835.029 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 8 sec latency 15835.309 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 9 sec latencStarting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3721216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7464960 1% /mnt/lustre test_70b fail mds1 2 times y 16835.924 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 10 sec latency 17837.196 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 11 sec latency 18837.436 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 12 sec latency 19837.694 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 13 sec latency 20838.373 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 14 sec latency 21840.331 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 15 sec latency 22846.841 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 16 sec latency 23848.775 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 17 sec latency 24849.198 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 18 sec latency 25849.644 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 19 sec latency 26850.074 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 20 sec latency 27850.903 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 21 sec latency 28851.096 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 22 sec latency 29851.285 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 23 sec latency 30855.568 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 24 sec latency 31855.793 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 25 sec latency 32857.819 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 26 sec latency 33858.059 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 27 sec latency 34858.276 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 28 sec latency 35858.640 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 29 sec latency 36859.033 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 30 sec latency 37859.468 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 31 sec latency 38860.290 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 32 sec latency 39860.547 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 33 sec latency 40864.484 ms oleg260-client.virtnet: 1 811 0.00 MB/sec execute 34 sec latency 41868.488 ms oleg260-client.virtnet: 1 821 0.00 MB/sec execute 35 sec latency 42626.365 ms oleg260-client.virtnet: 1 892 0.01 MB/sec execute 36 sec latency 222.694 ms oleg260-client.virtnet: 1 967 0.01 MB/sec execute 37 sec latency 129.939 ms oleg260-client.virtnet: 1 1002 0.01 MB/sec execute 38 sec latency 203.760 ms oleg260-client.virtnet: 1 1118 0.04 MB/sec execute 39 sec latency 113.968 ms oleg260-client.virtnet: 1 1178 0.04 MB/sec execute 40 sec latency 88.932 ms oleg260-client.virtnet: 1 1233 0.04 MB/sec execute 41 sec latency 88.246 ms oleg260-client.virtnet: 1 1278 0.04 MB/sec execute 42 sec latency 94.819 ms oleg260-client.virtnet: 1 1428 0.11 MB/sec execute 43 sec latency 83.800 ms oleg260-client.virtnet: 1 1512 0.10 MB/sec execute 44 sec latency 125.143 ms oleg260-client.virtnet: 1 1609 0.10 MB/sec execute 45 sec latency 87.071 ms oleg260-client.virtnet: 1 1662 0.10 MB/sec execute 46 sec latency 258.762 ms oleg260-client.virtnet: 1 1775 0.10 MB/sec execute 47 sec latency 69.408 ms oleg260-client.virtnet: 1 1855 0.10 MB/sec execute 48 sec latency 88.140 ms oleg260-client.virtnet: 1 1932 0.10 MB/sec execute 49 sec latency 170.377 ms oleg260-client.virtnet: 1 2002 0.10 MB/sec execute 50 sec latency 85.849 ms oleg260-client.virtnet: 1 2124 0.10 MB/sec execute 51 sec latency 89.561 ms oleg260-client.virtnet: 1 2191 0.10 MB/sec execute 52 sec latency 69.020 ms oleg260-client.virtnet: 1 2345 0.12 MB/sec execute 53 sec latency 19Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 1.692 ms oleg260-client.virtnet: 1 2389 0.12 MB/sec execute 54 sec latency 229.057 ms oleg260-client.virtnet: 1 2418 0.12 MB/sec execute 55 sec latency 328.821 ms oleg260-client.virtnet: 1 2479 0.13 MB/sec execute 56 sec latency 222.923 ms oleg260-client.virtnet: 1 2577 0.13 MB/sec execute 57 sec latency 96.162 ms oleg260-client.virtnet: 1 2592 0.13 MB/sec execute 58 sec latency 880.348 ms oleg260-client.virtnet: 1 2592 0.13 MB/sec execute 59 sec latency 1881.616 ms oleg260-client.virtnet: 1 2592 0.13 MB/sec execute 60 sec latency 2881.822 ms oleg260-client.virtnet: 1 2592 0.12 MB/sec execute 61 sec latency 3882.890 ms oleg260-client.virtnet: 1 2592 0.12 MB/sec execute 62 sec latency 4883.109 ms oleg260-client.virtnet: 1 2592 0.12 MB/sec execute 63 sec latency 5883.318 ms oleg260-client.virtnet: 1 2592 0.12 MB/sec execute 64 sec latency 6883.617 ms oleg260-client.virtnet: 1 2592 0.12 MB/sec execute 65 sec latency 7884.322 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 66 sec latency 8884.748 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 67 sec latency 9885.097 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 68 sec latency 10885.453 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 69 sec latency 11885.670 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 70 sec latency 12885.950 ms oleg260-client.virtnet: 1 2592 0.11 MB/sec execute 71 sec latency 13895.676 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 72 sec latency 14895.985 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 73 sec latency 15896.730 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 74 sec latency 16898.837 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 75 sec latency 17900.431 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 76 sec latency 18920.101 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 77 sec latency 19920.291 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 78 sec latency 20923.488 ms oleg260-client.virtnet: 1 2592 0.10 MB/sec execute 79 sec latency 21924.384 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 80 sec latency 22924.589 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 81 sec latency 23924.864 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 82 sec latency 24925.104 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 83 sec latency 25925.498 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 84 sec latency 26931.827 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 85 sec latency 27932.506 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 86 sec latency 28932.851 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 87 sec latency 29933.059 ms oleg260-client.virtnet: 1 2592 0.09 MB/sec execute 88 sec latency 30935.211 ms oleg260-client.virtnet: 1 2596 0.08 MB/sec execute 89 sec latency 31821.649 ms oleg260-client.virtnet: 1 2761 0.12 MB/sec execute 90 sec latency 130.296 ms oleg260-client.virtnet: 1 2896 0.12 MB/sec execute 91 sec latency 319.989 ms oleg260-client.virtnet: 1 2937 0.13 MB/sec execute 92 sec latency 318.834 ms oleg260-client.virtnet: 1 3000 0.13 MB/sec execute 93 sec latency 239.667 ms oleg260-client.virtnet: 1 3210 0.14 MB/sec execute 94 sec latency 148.730 ms oleg260-client.virtnet: 1 3347 0.15 MB/sec execute 95 sec latency 164.555 ms oleg260-client.virtnet: 1 3522 0.18 MB/sec execute 96 sec latency 72.266 ms oleg260-client.virtnet: 1 3624 0.18 MB/sec execute 97 sec latencyoleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 60.970 ms oleg260-client.virtnet: 1 3702 0.18 MB/sec execute 98 sec latency 96.229 ms oleg260-client.virtnet: 1 3785 0.18 MB/sec execute 99 sec latency 83.344 ms oleg260-client.virtnet: 1 3852 0.18 MB/sec execute 100 sec latency 140.346 ms oleg260-client.virtnet: 1 3924 0.18 MB/sec execute 101 sec latency 90.748 ms oleg260-client.virtnet: 1 3956 0.18 MB/sec execute 102 sec latency 147.862 ms oleg260-client.virtnet: 1 3995 0.18 MB/sec execute 103 sec latency 113.939 ms oleg260-client.virtnet: 1 4045 0.18 MB/sec execute 104 sec latency 103.063 ms oleg260-client.virtnet: 1 4079 0.18 MB/sec execute 105 sec latency 173.657 ms oleg260-client.virtnet: 1 4214 0.18 MB/sec execute 106 sec latency 81.104 ms oleg260-client.virtnet: 1 4248 0.18 MB/sec execute 107 sec latency 248.224 ms oleg260-client.virtnet: 1 4292 0.17 MB/sec execute 108 sec latency 320.736 ms oleg260-client.virtnet: 1 4318 0.17 MB/sec execute 109 sec latency 345.017 ms oleg260-client.virtnet: 1 4360 0.17 MB/sec execute 110 sec latency 390.856 ms oleg260-client.virtnet: 1 4379 0.17 MB/sec execute 111 sec latency 454.256 ms oleg260-client.virtnet: 1 4446 0.17 MB/sec execute 112 sec latency 271.792 ms oleg260-client.virtnet: 1 4521 0.17 MB/sec execute 113 sec latency 152.488 ms oleg260-client.virtnet: 1 4550 0.17 MB/sec execute 114 sec latency 300.080 ms oleg260-client.virtnet: 1 4629 0.18 MB/sec execute 115 sec latency 107.903 ms oleg260-client.virtnet: 1 4698 0.18 MB/sec execute 116 sec latency 99.690 ms oleg260-client.virtnet: 1 4781 0.18 MB/sec execute 117 sec latency 75.437 ms oleg260-client.virtnet: 1 4843 0.18 MB/sec execute 118 sec latency 69.822 ms oleg260-client.virtnet: 1 5013 0.20 MB/sec execute 119 sec latency 73.940 ms oleg260-client.virtnet: 1 cleanup 120 sec oleg260-client.virtnet: 1 cleanup 121 sec oleg260-client.virtnet: 0 cleanup 122 sec oleg260-client.virtnet: oleg260-client.virtnet: Operation Count AvgLat MaxLat oleg260-client.virtnet: ---------------------------------------- oleg260-client.virtnet: NTCreateX 728 33.902 173.641 oleg260-client.virtnet: Close 544 9.748 79.357 oleg260-client.virtnet: Rename 34 42.485 130.263 oleg260-client.virtnet: Unlink 139 319.881 42626.343 oleg260-client.virtnet: Qpathinfo 678 56.125 31821.593 oleg260-client.virtnet: Qfileinfo 115 0.753 14.077 oleg260-client.virtnet: Qfsinfo 122 1.473 37.216 oleg260-client.virtnet: Sfileinfo 75 28.040 77.901 oleg260-client.virtnet: Find 257 2.288 122.399 oleg260-client.virtnet: WriteX 368 5.428 51.558 oleg260-client.virtnet: ReadX 1154 0.400 113.943 oleg260-client.virtnet: LockX 2 9.088 12.522 oleg260-client.virtnet: UnlockX 2 20.273 21.367 oleg260-client.virtnet: Flush 50 170.707 454.240 oleg260-client.virtnet: oleg260-client.virtnet: Throughput 0.198394 MB/sec 1 clients 1 procs max_latency=42626.365 ms oleg260-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg260-client.virtnet at Tue Apr 14 18:51:40 EDT 2026 with return code 0 oleg260-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg260-client.virtnet oleg260-client.virtnet: /mnt/lustre/d70b.replay-single/oleg260-client.virtnet /mnt/lustre/d70b.replay-single/oleg260-client.virtnet oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg260-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg260-client.virtnet: removed directory 'clients/client0' oleg260-client.virtnet: removed directory 'clients' oleg260-client.virtnet: removed 'client.txt' oleg260-client.virtnet: /mnt/lustre/d70b.replay-single/oleg260-client.virtnet oleg260-client.virtnet: dbench successfully finished PASS 70b (156s) == replay-single test 70c: tar 1mdts recovery ============ 18:51:47 (1776207107) Starting client oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar 134307 tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6784 2201472 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3588096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3538944 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 27648 7127040 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (228s) == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 18:55:35 (1776207335) SKIP: replay-single test_70d needs >= 2 MDTs SKIP 70d (3s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 18:55:38 (1776207338) SKIP: replay-single test_70e needs >= 2 MDTs SKIP 70e (4s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 18:55:42 (1776207342) mount clients oleg260-client.virtnet ... Starting client oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 8192 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7502848 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 8192 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7518208 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg260-client.virtnet' ... ldlm.namespaces.MGC192.168.202.160@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa09d06973800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa09d06973800.lru_size=clear PASS 70f (118s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 18:57:40 (1776207460) SKIP: replay-single test_71a needs >= 2 MDTs SKIP 71a (3s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 18:57:44 (1776207464) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (59s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 18:58:42 (1776207522) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (60s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 18:59:42 (1776207582) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg260-server Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (66s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:00:48 (1776207648) SKIP: replay-single test_80a needs >= 2 MDTs SKIP 80a (3s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:00:51 (1776207651) SKIP: replay-single test_80b needs >= 2 MDTs SKIP 80b (3s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:00:54 (1776207654) SKIP: replay-single test_80c needs >= 2 MDTs SKIP 80c (3s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:00:57 (1776207657) SKIP: replay-single test_80d needs >= 2 MDTs SKIP 80d (4s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:01:01 (1776207661) SKIP: replay-single test_80e needs >= 2 MDTs SKIP 80e (4s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:01:05 (1776207665) SKIP: replay-single test_80f needs >= 2 MDTs SKIP 80f (4s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:01:09 (1776207669) SKIP: replay-single test_80g needs >= 2 MDTs SKIP 80g (4s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:01:13 (1776207673) SKIP: replay-single test_80h needs >= 2 MDTs SKIP 80h (4s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:01:17 (1776207677) SKIP: replay-single test_81a needs >= 2 MDTs SKIP 81a (4s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:01:21 (1776207681) SKIP: replay-single test_81b needs >= 2 MDTs SKIP 81b (4s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:01:25 (1776207685) SKIP: replay-single test_81c needs >= 2 MDTs SKIP 81c (3s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:01:28 (1776207688) SKIP: replay-single test_81d needs >= 2 MDTs SKIP 81d (3s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:01:32 (1776207692) SKIP: replay-single test_81e needs >= 2 MDTs SKIP 81e (4s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:01:35 (1776207695) SKIP: replay-single test_81f needs >= 2 MDTs SKIP 81f (3s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:01:39 (1776207699) SKIP: replay-single test_81g needs >= 2 MDTs SKIP 81g (3s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:01:43 (1776207703) SKIP: replay-single test_81h needs >= 2 MDTs SKIP 81h (3s) == replay-single test 84a: stale open during export disconnect ========================================================== 19:01:46 (1776207706) fail_loc=0x80000144 total: 1 open/close in 0.04 seconds: 25.76 ops/second oleg260-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg260-client: oleg260-client: ssh exited with exit code 5 PASS 84a (12s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:01:58 (1776207718) before recovery: unused locks count = 301 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (62s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:03:00 (1776207780) before recovery: unused locks count = 100 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (59s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:04:00 (1776207840) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:) Stopping client oleg260-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre Started clients oleg260-client.virtnet: 192.168.202.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 86 (27s) == replay-single test 87a: write replay ================== 19:04:26 (1776207866) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.38657 s, 21.7 MB/s Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.229122 s, 36.6 MB/s PASS 87a (53s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:05:19 (1776207919) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.464658 s, 18.1 MB/s 8+0 records in 8+0 records out 8 bytes copied, 0.0205429 s, 0.4 kB/s Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes copied, 0.0317492 s, 2.3 kB/s PASS 87b (57s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:06:16 (1776207976) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3757056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 15360 7523328 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3757056 1% /mnt/lustre[OST:0] R lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 15360 7523328 1% /mnt/lustre before test: last_id = 7009, next_id = 6978 Creating to objid 7009 on ost lustre-OST0000... total: 33 open/close in 0.79 seconds: 42.02 ops/second total: 8 open/close in 0.12 seconds: 68.08 ops/second before recovery: last_id = 7009, next_id = 6978 Stopping /mnt/lustre-mds1 (opts:) on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server Failover mds1 to oleg260-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg260-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 7041, next_id = 7010 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.180291 s, 2.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.157171 s, 3.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.204817 s, 2.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.10361 s, 5.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.148978 s, 3.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.130434 s, 4.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.142823 s, 3.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.063467 s, 8.3 MB/s -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6978 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6979 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6980 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6981 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6982 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6983 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6984 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6985 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6986 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6987 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6988 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6989 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6990 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6991 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6992 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6993 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6994 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6995 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6996 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6997 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6998 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-6999 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7000 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7001 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7002 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7003 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7004 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7005 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7006 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7007 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7008 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7009 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7010 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7011 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7012 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7013 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7014 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7015 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7016 -rw-r--r-- 1 root root 0 Apr 14 19:06 /mnt/lustre/d88.replay-single/f-7017 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7021 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7022 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7023 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7024 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7025 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7026 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7027 -rw-r--r-- 1 root root 524288 Apr 14 19:07 /mnt/lustre/d88.replay-single/f-7028 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0855309 s, 6.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0770473 s, 6.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0501054 s, 10.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.107414 s, 4.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.105888 s, 5.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.168383 s, 3.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.124006 s, 4.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0937924 s, 5.6 MB/s PASS 88 (99s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 19:07:55 (1776208075) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 1.10759 s, 37.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg260-server Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg260-client.virtnet: -o user_xattr,flock oleg260-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete free_before: 7518208 free_after: 7518208 PASS 89 (168s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 19:10:43 (1776208243) Waiting 90s for '2' Waiting 80s for '2' Waiting 70s for '2' Waiting 60s for '2' Waiting 50s for '2' Waiting 40s for '2' Waiting 30s for '2' Waiting 20s for '2' Update not seen after 90s: want '2' got '1' replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6459:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:3525:wait_osts_up() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3350:test_90() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6690:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3425:main() Dumping lctl log to /tmp/testlogs//replay-single.test_90.*.1776208338.log rsync: chown "/tmp/testlogs/.replay-single.test_90.debug_log.oleg260-server.1776208338.log.oGf2up" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_90.dmesg.oleg260-server.1776208338.log.VpO4ku" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 90 (99s) == replay-single test 93a: replay + reconnect ============ 19:12:28 (1776208348) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00778272 s, 132 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg260-server Stopping /mnt/lustre-ost1 (opts:) on oleg260-server reboot facets: ost1 Failover ost1 to oleg260-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (80s) == replay-single test 93b: replay + reconnect on mds ===== 19:13:49 (1776208429) total: 20 open/close in 0.27 seconds: 74.94 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server reboot facets: mds1 Failover mds1 to oleg260-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (128s) == replay-single test complete, duration 7511 sec ======== 19:15:57 (1776208557) replay-single: FAIL: test_90 wait_update OSTs up on MDT0000 failed