-----============= acceptance-small: replay-single ============----- Tue Dec 17 05:11:36 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg254-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 110f 131b 59 36 === replay-single: start setup 05:11:45 (1734430305) === oleg254-client.virtnet: executing check_config_client /mnt/lustre oleg254-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg254-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3ae0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3ae0000.idle_timeout=debug disable quota as required oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 05:12:02 (1734430322) === == replay-single test 0a: empty replay =================== 05:12:02 (1734430322) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3072 7210968 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:12:09 (1734430329) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:12:25 (1734430345) targets are mounted 05:12:25 (1734430345) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (33s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 05:12:35 (1734430355) Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 05:12:37 (1734430357) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 05:12:54 (1734430374) targets are mounted 05:12:54 (1734430374) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.24 seconds: 82.53 ops/second - unlinked 0 (time 1734430379 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (26s) == replay-single test 0c: check replay-barrier =========== 05:13:01 (1734430381) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:13:07 (1734430387) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:13:23 (1734430403) targets are mounted 05:13:23 (1734430403) facet_failover done Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (96s) == replay-single test 0d: expired recovery with no clients ========================================================== 05:14:38 (1734430478) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:14:43 (1734430483) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:14:59 (1734430499) targets are mounted 05:14:59 (1734430499) facet_failover done Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre PASS 0d (95s) == replay-single test 1: simple create =================== 05:16:12 (1734430572) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:16:19 (1734430579) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:16:36 (1734430596) targets are mounted 05:16:36 (1734430596) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (31s) == replay-single test 2a: touch ========================== 05:16:43 (1734430603) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:16:48 (1734430608) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:17:04 (1734430624) targets are mounted 05:17:04 (1734430624) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (29s) == replay-single test 2b: touch ========================== 05:17:12 (1734430632) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:17:20 (1734430640) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:17:36 (1734430656) targets are mounted 05:17:36 (1734430656) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (32s) == replay-single test 2c: setstripe replay =============== 05:17:44 (1734430664) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:17:50 (1734430670) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:18:06 (1734430686) targets are mounted 05:18:06 (1734430686) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (31s) == replay-single test 2d: setdirstripe replay ============ 05:18:15 (1734430695) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:18:21 (1734430701) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:18:37 (1734430717) targets are mounted 05:18:37 (1734430717) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (30s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 05:18:45 (1734430725) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:18:52 (1734430732) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:19:08 (1734430748) targets are mounted 05:19:08 (1734430748) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (31s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 05:19:16 (1734430756) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:19:21 (1734430761) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:19:37 (1734430777) targets are mounted 05:19:37 (1734430777) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (28s) == replay-single test 3b: replay failed open -ENOMEM ===== 05:19:44 (1734430784) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:19:50 (1734430790) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:20:06 (1734430806) targets are mounted 05:20:06 (1734430806) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (29s) == replay-single test 3c: replay failed open -ENOMEM ===== 05:20:13 (1734430813) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:20:19 (1734430819) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:20:33 (1734430833) targets are mounted 05:20:33 (1734430833) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (28s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 05:20:41 (1734430841) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:20:46 (1734430846) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:21:01 (1734430861) targets are mounted 05:21:01 (1734430861) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (28s) == replay-single test 4b: |x| rm 10 files ================ 05:21:09 (1734430869) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210644 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:21:16 (1734430876) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:21:30 (1734430890) targets are mounted 05:21:30 (1734430890) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (29s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 05:21:38 (1734430898) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:21:43 (1734430903) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:21:58 (1734430918) targets are mounted 05:21:58 (1734430918) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (37s) == replay-single test 6a: mkdir + contained create ======= 05:22:15 (1734430935) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:22:20 (1734430940) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:22:36 (1734430956) targets are mounted 05:22:36 (1734430956) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (31s) == replay-single test 6b: |X| rmdir ====================== 05:22:46 (1734430966) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:22:51 (1734430971) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:23:07 (1734430987) targets are mounted 05:23:07 (1734430987) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (28s) == replay-single test 7: mkdir |X| contained create ====== 05:23:14 (1734430994) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:23:19 (1734430999) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:23:35 (1734431015) targets are mounted 05:23:35 (1734431015) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (29s) == replay-single test 8: creat open |X| close ============ 05:23:43 (1734431023) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:23:48 (1734431028) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:24:04 (1734431044) targets are mounted 05:24:04 (1734431044) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (28s) == replay-single test 9: |X| create (same inum/gen) ====== 05:24:11 (1734431051) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:24:16 (1734431056) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:24:32 (1734431072) targets are mounted 05:24:32 (1734431072) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (29s) == replay-single test 10: create |X| rename unlink ======= 05:24:41 (1734431081) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:24:47 (1734431087) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:04 (1734431104) targets are mounted 05:25:04 (1734431104) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (33s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 05:25:13 (1734431113) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre new old Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:25:18 (1734431118) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:35 (1734431135) targets are mounted 05:25:36 (1734431136) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (31s) == replay-single test 12: open, unlink |X| close ========= 05:25:44 (1734431144) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:25:50 (1734431150) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:26:06 (1734431166) targets are mounted 05:26:06 (1734431166) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (30s) == replay-single test 13: open chmod 0 |x| write close === 05:26:14 (1734431174) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:26:18 (1734431178) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:26:34 (1734431194) targets are mounted 05:26:34 (1734431194) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (28s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 05:26:42 (1734431202) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:26:47 (1734431207) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:27:02 (1734431222) targets are mounted 05:27:02 (1734431222) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (28s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 05:27:10 (1734431230) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:27:15 (1734431235) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:27:30 (1734431250) targets are mounted 05:27:30 (1734431250) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (29s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 05:27:39 (1734431259) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:27:44 (1734431264) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:28:01 (1734431281) targets are mounted 05:28:01 (1734431281) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (31s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 05:28:10 (1734431290) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:28:15 (1734431295) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:28:32 (1734431312) targets are mounted 05:28:32 (1734431312) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (30s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 05:28:40 (1734431320) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 pid: 23050 will close Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:28:47 (1734431327) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:29:04 (1734431344) targets are mounted 05:29:04 (1734431344) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (32s) == replay-single test 19: mcreate, open, write, rename === 05:29:12 (1734431352) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre old Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:29:18 (1734431358) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:29:36 (1734431376) targets are mounted 05:29:36 (1734431376) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (31s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 05:29:44 (1734431384) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210912 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:29:49 (1734431389) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:30:06 (1734431406) targets are mounted 05:30:06 (1734431406) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (31s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 05:30:15 (1734431415) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x280000401 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown 8473+0 records in 8472+0 records out 34701312 bytes (35 MB) copied, 1.85119 s, 18.7 MB/s pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:30:22 (1734431422) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:30:37 (1734431437) targets are mounted 05:30:37 (1734431437) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg254-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 before 3128, after 3128 PASS 20b (35s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 05:30:50 (1734431450) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 -rw-r--r-- 1 root root 1 Dec 17 05:30 /mnt/lustre/f20c.replay-single pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 PASS 20c (5s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 05:30:56 (1734431456) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210880 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:31:01 (1734431461) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:31:18 (1734431478) targets are mounted 05:31:18 (1734431478) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (32s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 05:31:27 (1734431487) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210876 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:31:32 (1734431492) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:31:49 (1734431509) targets are mounted 05:31:49 (1734431509) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (30s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 05:31:57 (1734431517) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210876 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:32:03 (1734431523) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:18 (1734431538) targets are mounted 05:32:18 (1734431538) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (30s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 05:32:27 (1734431547) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210876 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:32:32 (1734431552) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:47 (1734431567) targets are mounted 05:32:47 (1734431567) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (29s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 05:32:56 (1734431576) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:33:01 (1734431581) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:17 (1734431597) targets are mounted 05:33:17 (1734431597) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (29s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 05:33:25 (1734431605) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:33:30 (1734431610) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:46 (1734431626) targets are mounted 05:33:46 (1734431626) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (29s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 05:33:54 (1734431634) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:33:59 (1734431639) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:15 (1734431655) targets are mounted 05:34:15 (1734431655) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (29s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 05:34:23 (1734431663) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:34:28 (1734431668) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:43 (1734431683) targets are mounted 05:34:43 (1734431683) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (27s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 05:34:50 (1734431690) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:34:56 (1734431696) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:12 (1734431712) targets are mounted 05:35:12 (1734431712) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (29s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 05:35:19 (1734431719) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:35:24 (1734431724) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:39 (1734431739) targets are mounted 05:35:39 (1734431739) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (27s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 05:35:46 (1734431746) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:35:50 (1734431750) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:06 (1734431766) targets are mounted 05:36:06 (1734431766) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (28s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 05:36:14 (1734431774) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 PASS 32 (5s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 05:36:19 (1734431779) total: 10 open/close in 0.09 seconds: 107.94 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.06 seconds: 164.74 ops/second PASS 33a (21s) == replay-single test 33b: test fid seq allocation ======= 05:36:40 (1734431800) fail_loc=0x1311 total: 10 open/close in 0.07 seconds: 135.29 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.08 seconds: 119.85 ops/second PASS 33b (20s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 05:37:00 (1734431820) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (22s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 05:37:22 (1734431842) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 first stat failed: 5 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (18s) SKIP: replay-single test_36 skipping ALWAYS excluded test 36 == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 05:37:41 (1734431861) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1984 1285704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1285900 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (20s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 05:38:01 (1734431881) total: 800 open/close in 4.55 seconds: 175.65 ops/second - unlinked 0 (time 1734431888 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:38:13 (1734431893) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:38:28 (1734431908) targets are mounted 05:38:28 (1734431908) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1734431915 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (41s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 05:38:42 (1734431922) total: 800 open/close in 4.31 seconds: 185.46 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2112 1285576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210908 1% /mnt/lustre - unlinked 0 (time 1734431931 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:38:54 (1734431934) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:39:08 (1734431948) targets are mounted 05:39:08 (1734431948) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1734431956 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (40s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 05:39:23 (1734431963) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00213923 s, 1.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00773578 s, 529 kB/s PASS 41 (3s) == replay-single test 42: recovery after ost failure ===== 05:39:26 (1734431966) total: 800 open/close in 3.62 seconds: 221.27 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1734431974 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second debug=-1 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 05:39:38 (1734431978) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 05:39:53 (1734431993) targets are mounted 05:39:53 (1734431993) facet_failover done wait for MDS to timeout and recover - unlinked 0 (time 1734432034 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (71s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 05:40:37 (1734432037) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2192 1285496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:40:41 (1734432041) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:40:55 (1734432055) targets are mounted 05:40:55 (1734432055) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (36s) == replay-single test 44a: race in target handle connect ========================================================== 05:41:13 (1734432073) at_max=40 1 of 10 (1734432074) service : cur 5 worst 5 (at 1734430252, 1822s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 2 of 10 (1734432075) service : cur 5 worst 5 (at 1734430252, 1823s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 3 of 10 (1734432081) service : cur 5 worst 5 (at 1734430252, 1828s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 4 of 10 (1734432086) service : cur 5 worst 5 (at 1734430252, 1834s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 5 of 10 (1734432092) service : cur 5 worst 5 (at 1734430252, 1839s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 6 of 10 (1734432097) service : cur 5 worst 5 (at 1734430252, 1845s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 7 of 10 (1734432103) service : cur 5 worst 5 (at 1734430252, 1851s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 8 of 10 (1734432108) service : cur 5 worst 5 (at 1734430252, 1856s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 9 of 10 (1734432114) service : cur 5 worst 5 (at 1734430252, 1862s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 10 of 10 (1734432119) service : cur 5 worst 5 (at 1734430252, 1867s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (54s) == replay-single test 44b: race in target handle connect ========================================================== 05:42:07 (1734432127) 1 of 10 (1734432127) service : cur 5 worst 5 (at 1734430252, 1875s ago) 5 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 2 of 10 (1734432168) service : cur 40 worst 40 (at 1734432167, 1s ago) 40 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 3 of 10 (1734432189) service : cur 40 worst 40 (at 1734432167, 21s ago) 40 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 4 of 10 (1734432209) service : cur 40 worst 40 (at 1734432167, 42s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 5 of 10 (1734432230) service : cur 40 worst 40 (at 1734432167, 62s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 6 of 10 (1734432250) service : cur 40 worst 40 (at 1734432167, 83s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 7 of 10 (1734432271) service : cur 40 worst 40 (at 1734432167, 104s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 8 of 10 (1734432291) service : cur 40 worst 40 (at 1734432167, 124s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 9 of 10 (1734432312) service : cur 40 worst 40 (at 1734432167, 145s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 10 of 10 (1734432332) service : cur 40 worst 40 (at 1734432167, 165s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.154@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre PASS 44b (227s) == replay-single test 44c: race in target handle connect ========================================================== 05:45:54 (1734432354) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre total: 100 create in 0.40 seconds: 252.99 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 first stat failed: 5 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:46:12 (1734432372) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:46:25 (1734432385) targets are mounted 05:46:25 (1734432385) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (38s) == replay-single test 45: Handle failed close ============ 05:46:32 (1734432392) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 /mnt/lustre/f45.replay-single has type file OK PASS 45 (2s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 05:46:34 (1734432394) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:46:52 (1734432412) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:47:05 (1734432425) targets are mounted 05:47:05 (1734432425) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (39s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 05:47:13 (1734432433) total: 20 open/close in 0.07 seconds: 283.78 ops/second Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 05:47:14 (1734432434) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 05:47:28 (1734432448) targets are mounted 05:47:28 (1734432448) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.05 seconds: 372.79 ops/second - unlinked 0 (time 1734432513 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (81s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 05:48:34 (1734432514) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre total: 20 open/close in 0.07 seconds: 304.96 ops/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:48:37 (1734432517) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:48:52 (1734432532) targets are mounted 05:48:52 (1734432532) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.13 seconds: 150.25 ops/second - unlinked 0 (time 1734432595 ; total 0 ; last 0) total: 40 unlinks in 1 seconds: 40.000000 unlinks/second PASS 48 (83s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 05:49:57 (1734432597) PASS 50 (7s) == replay-single test 52: time out lock replay (3764) ==== 05:50:04 (1734432604) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7604 fail_loc=0x80000157 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:50:06 (1734432606) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:50:20 (1734432620) targets are mounted 05:50:20 (1734432620) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (79s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 05:51:23 (1734432683) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:51:28 (1734432688) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:51:42 (1734432702) targets are mounted 05:51:42 (1734432702) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (27s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 05:51:50 (1734432710) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:51:55 (1734432715) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:52:09 (1734432729) targets are mounted 05:52:09 (1734432729) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (26s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 05:52:16 (1734432736) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:52:21 (1734432741) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:52:35 (1734432755) targets are mounted 05:52:35 (1734432755) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (26s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 05:52:42 (1734432762) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:52:45 (1734432765) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:52:58 (1734432778) targets are mounted 05:52:58 (1734432778) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (23s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 05:53:05 (1734432785) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:53:10 (1734432790) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:53:24 (1734432804) targets are mounted 05:53:24 (1734432804) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (27s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 05:53:32 (1734432812) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:53:37 (1734432817) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:53:51 (1734432831) targets are mounted 05:53:51 (1734432831) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (26s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 05:53:58 (1734432838) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:54:03 (1734432843) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:17 (1734432857) targets are mounted 05:54:17 (1734432857) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (26s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 05:54:24 (1734432864) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:54:30 (1734432870) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:44 (1734432884) targets are mounted 05:54:44 (1734432884) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (27s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 05:54:51 (1734432891) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (62s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 05:55:53 (1734432953) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2128 1285560 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:55:57 (1734432957) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:56:11 (1734432971) targets are mounted 05:56:11 (1734432971) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (35s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 05:56:28 (1734432988) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2128 1285560 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:56:32 (1734432992) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:56:46 (1734433006) targets are mounted 05:56:46 (1734433006) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg254-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (29s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 05:56:57 (1734433017) fail_loc=0x8000012c total: 2500 open/close in 6.15 seconds: 406.62 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:57:08 (1734433028) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:57:22 (1734433042) targets are mounted 05:57:22 (1734433042) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1734433054 ; total 0 ; last 0) total: 2500 unlinks in 2 seconds: 1250.000000 unlinks/second PASS 58a (41s) == replay-single test 58b: test replay of setxattr op ==== 05:57:38 (1734433058) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 05:57:42 (1734433062) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 05:57:56 (1734433076) targets are mounted 05:57:56 (1734433076) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) oleg254-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (27s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 05:58:05 (1734433085) Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) PASS 58c (128s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 06:00:14 (1734433214) total: 200 open/close in 0.38 seconds: 530.66 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2260 1285428 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1734433217 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:00:18 (1734433218) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:00:33 (1734433233) targets are mounted 06:00:33 (1734433233) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1734433239 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second PASS 60 (27s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 06:00:41 (1734433241) total: 800 open/close in 1.72 seconds: 464.87 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1734433247 ; total 0 ; last 0) total: 800 unlinks in 1 seconds: 800.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:00:50 (1734433250) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:01:05 (1734433265) targets are mounted 06:01:05 (1734433265) facet_failover done Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:01:16 (1734433276) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:01:29 (1734433289) targets are mounted 06:01:29 (1734433289) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (83s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 06:02:04 (1734433324) fail_loc=0x8000013a Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:02:06 (1734433326) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:02:19 (1734433339) targets are mounted 06:02:19 (1734433339) facet_failover done Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:02:30 (1734433350) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:02:43 (1734433363) targets are mounted 06:02:43 (1734433363) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00356415 s, 1.1 MB/s PASS 61b (46s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 06:02:50 (1734433370) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:03:02 (1734433382) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:03:16 (1734433396) targets are mounted 06:03:16 (1734433396) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (31s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 06:03:21 (1734433401) Stopping /mnt/lustre-mds1 (opts:) on oleg254-server fail_loc=0x80000605 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (8s) == replay-single test 62: don't mis-drop resent replay === 06:03:29 (1734433409) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2332 1285356 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3140 7210900 1% /mnt/lustre total: 25 open/close in 0.06 seconds: 401.14 ops/second fail_loc=0x80000707 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:03:35 (1734433415) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:03:50 (1734433430) targets are mounted 06:03:50 (1734433430) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1734433495 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second PASS 62 (88s) == replay-single test 65a: AT: verify early replies ====== 06:04:57 (1734433497) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:2.0:1734433526.757360:0:7484:0:(client.c:554:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a7dbf800 x1818681875413376/t0(0) o101->lustre-MDT0000-mdc-ffff8800b382a800@192.168.202.154@tcp:12/10 lens 664/66320 e 1 to 0 dl 1734433561 ref 2 fl Rpc:PQr/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 portal 12 : cur 36 worst 40 (at 1734432168, 1364s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1734430569, 2963s ago) 5 5 0 0 portal 23 : cur 5 worst 5 (at 1734430569, 2963s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1734430573, 2959s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1734430668, 2864s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1734430867, 2665s ago) 5 0 0 0 portal 13 : cur 5 worst 5 (at 1734431085, 2447s ago) 5 5 0 0 portal 12 : cur 11 worst 40 (at 1734432168, 1373s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1734430569, 2972s ago) 5 5 0 0 portal 23 : cur 5 worst 5 (at 1734430569, 2972s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1734430573, 2968s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1734430668, 2873s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1734430867, 2674s ago) 5 0 0 0 portal 13 : cur 5 worst 5 (at 1734431085, 2456s ago) 5 5 0 0 PASS 65a (45s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 06:05:42 (1734433542) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:3.0:1734433571.413277:0:2010:0:(client.c:554:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a7a93480 x1818681875433088/t0(0) o4->lustre-OST0000-osc-ffff8800b382a800@192.168.202.154@tcp:6/4 lens 4584/448 e 1 to 0 dl 1734433606 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 portal 28 : cur 5 worst 5 (at 1734430569, 3008s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1734430571, 3006s ago) 5 5 5 5 portal 6 : cur 36 worst 36 (at 1734433576, 1s ago) 36 5 5 0 portal 17 : cur 5 worst 5 (at 1734430873, 2704s ago) 5 0 0 5 PASS 65b (36s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 06:06:18 (1734433578) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1734432168, 1432s ago) 5 40 40 40 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1734432168, 1438s ago) 5 40 40 40 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 36 worst 40 (at 1734432168, 1448s ago) 36 40 40 40 fail_loc=0 portal 12 : cur 5 worst 40 (at 1734432168, 1458s ago) 36 40 40 40 Current MDT timeout 5, worst 40 PASS 66a (49s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 06:07:07 (1734433627) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (93s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 06:08:40 (1734433720) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (73s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 06:09:53 (1734433793) at_history=8 at_history=8 Creating to objid 5057 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.08 seconds: 220.12 ops/second Connected clients: oleg254-client.virtnet oleg254-client.virtnet service : cur 5 worst 5 (at 1734430265, 3552s ago) 1 1 1 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg254-client.virtnet oleg254-client.virtnet service : cur 5 worst 5 (at 1734430265, 3554s ago) 1 1 1 1 0 osc reconnect attempts on 2nd slow PASS 67b (28s) == replay-single test 68: AT: verify slowing locks ======= 06:10:21 (1734433821) at_history=8 at_history=8 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1973: $ldlm_enqueue_min: ambiguous redirect fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1988: $ldlm_enqueue_min: ambiguous redirect PASS 68 (69s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 06:11:31 (1734433891) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (0s) == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:11:31 (1734433891) Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg254-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13797 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3593664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3580872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7174536 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:11:40 (1734433900) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:11:53 (1734433913) targets are mounted 06:11:53 (1734433913) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1285268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1736 3594108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26440 3568812 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28176 7162920 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:12:05 (1734433925) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server oleg254-client.virtnet: looking for dbench program oleg254-client.virtnet: /usr/bin/dbench oleg254-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg254-client.virtnet' oleg254-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg254-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg254-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Dec 17 06:11:32 EST 2024 oleg254-client.virtnet: waiting for dbench pid 13821 oleg254-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg254-client.virtnet: oleg254-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg254-client.virtnet: failed to create barrier semaphore oleg254-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg254-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg254-client.virtnet: releasing clients oleg254-client.virtnet: 1 358 12.78 MB/sec warmup 1 sec latency 19.703 ms oleg254-client.virtnet: 1 765 10.68 MB/sec warmup 2 sec latency 21.238 ms oleg254-client.virtnet: 1 1086 7.64 MB/sec warmup 3 sec latency 18.103 ms oleg254-client.virtnet: 1 1587 6.54 MB/sec warmup 4 sec latency 16.147 ms oleg254-client.virtnet: 1 1995 5.30 MB/sec warmup 5 sec latency 17.696 ms oleg254-client.virtnet: 1 2454 4.69 MB/sec warmup 6 sec latency 15.867 ms oleg254-client.virtnet: 1 3210 4.90 MB/sec warmup 7 sec latency 18.391 ms oleg254-client.virtnet: 1 3814 4.97 MB/sec warmup 8 sec latency 17.519 ms oleg254-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 115.566 ms oleg254-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 1115.810 ms oleg254-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 2115.994 ms oleg254-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 3116.190 ms oleg254-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 4116.726 ms oleg254-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 5116.910 ms oleg254-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 6117.122 ms oleg254-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 7117.335 ms oleg254-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 8117.465 ms oleg254-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 9117.592 ms oleg254-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 10117.723 ms oleg254-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 11117.859 ms oleg254-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 12118.034 ms oleg254-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 13118.151 ms oleg254-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 14118.292 ms oleg254-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 15118.441 ms oleg254-client.virtnet: 1 4302 1.62 MB/sec warmup 25 sec latency 15689.450 ms oleg254-client.virtnet: 1 4787 1.62 MB/sec warmup 26 sec latency 11.221 ms oleg254-client.virtnet: 1 5278 1.68 MB/sec warmup 27 sec latency 16.894 ms oleg254-client.virtnet: 1 5907 1.68 MB/sec warmup 28 sec latency 12.772 ms oleg254-client.virtnet: 1 6741 1.84 MB/sec warmup 29 sec latency 11.845 ms oleg254-client.virtnet: 1 7433 1.96 MB/sec warmup 30 sec latency 10.537 ms oleg254-client.virtnet: 1 8014 1.93 MB/sec warmup 31 sec latency 11.236 ms oleg254-client.virtnet: 1 8722 2.01 MB/sec warmup 32 sec latency 50.273 ms oleg254-client.virtnet: 1 8722 1.95 MB/sec warmup 33 sec latency 1050.452 mFailover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:12:19 (1734433939) targets are mounted 06:12:19 (1734433939) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12164 3593688 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3566696 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7160384 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:12:31 (1734433951) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:12:45 (1734433965) targets are mounted 06:12:45 (1734433965) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec s oleg254-client.virtnet: 1 8722 1.89 MB/sec warmup 34 sec latency 2050.637 ms oleg254-client.virtnet: 1 8722 1.84 MB/sec warmup 35 sec latency 3050.784 ms oleg254-client.virtnet: 1 8722 1.79 MB/sec warmup 36 sec latency 4050.951 ms oleg254-client.virtnet: 1 8722 1.74 MB/sec warmup 37 sec latency 5051.105 ms oleg254-client.virtnet: 1 8722 1.69 MB/sec warmup 38 sec latency 6051.264 ms oleg254-client.virtnet: 1 8722 1.65 MB/sec warmup 39 sec latency 7051.419 ms oleg254-client.virtnet: 1 8722 1.61 MB/sec warmup 40 sec latency 8051.575 ms oleg254-client.virtnet: 1 8722 1.57 MB/sec warmup 41 sec latency 9051.740 ms oleg254-client.virtnet: 1 8722 1.53 MB/sec warmup 42 sec latency 10051.906 ms oleg254-client.virtnet: 1 8722 1.50 MB/sec warmup 43 sec latency 11052.089 ms oleg254-client.virtnet: 1 8722 1.46 MB/sec warmup 44 sec latency 12052.219 ms oleg254-client.virtnet: 1 8722 1.43 MB/sec warmup 45 sec latency 13052.358 ms oleg254-client.virtnet: 1 8722 1.40 MB/sec warmup 46 sec latency 14052.533 ms oleg254-client.virtnet: 1 8722 1.37 MB/sec warmup 47 sec latency 15053.162 ms oleg254-client.virtnet: 1 8722 1.34 MB/sec warmup 48 sec latency 16053.315 ms oleg254-client.virtnet: 1 8722 1.31 MB/sec warmup 49 sec latency 17053.472 ms oleg254-client.virtnet: 1 8722 1.29 MB/sec warmup 50 sec latency 18053.643 ms oleg254-client.virtnet: 1 9018 1.27 MB/sec warmup 51 sec latency 18424.197 ms oleg254-client.virtnet: 1 9493 1.27 MB/sec warmup 52 sec latency 15.717 ms oleg254-client.virtnet: 1 10111 1.35 MB/sec warmup 53 sec latency 16.194 ms oleg254-client.virtnet: 1 10858 1.44 MB/sec warmup 54 sec latency 15.422 ms oleg254-client.virtnet: 1 11202 1.42 MB/sec warmup 55 sec latency 19.342 ms oleg254-client.virtnet: 1 11759 1.43 MB/sec warmup 56 sec latency 11.906 ms oleg254-client.virtnet: 1 12372 1.47 MB/sec warmup 57 sec latency 12.671 ms oleg254-client.virtnet: 1 13034 1.47 MB/sec warmup 58 sec latency 9.653 ms oleg254-client.virtnet: 1 14014 1.57 MB/sec warmup 59 sec latency 10.816 ms oleg254-client.virtnet: 1 15280 1.98 MB/sec execute 1 sec latency 9.147 ms oleg254-client.virtnet: 1 15932 2.67 MB/sec execute 2 sec latency 9.074 ms oleg254-client.virtnet: 1 16602 2.36 MB/sec execute 3 sec latency 8.883 ms oleg254-client.virtnet: 1 17722 4.36 MB/sec execute 4 sec latency 8.671 ms oleg254-client.virtnet: 1 18234 3.82 MB/sec execute 5 sec latency 10.944 ms oleg254-client.virtnet: 1 18776 3.48 MB/sec execute 6 sec latency 11.430 ms oleg254-client.virtnet: 1 19384 3.48 MB/sec execute 7 sec latency 11.803 ms oleg254-client.virtnet: 1 20069 3.19 MB/sec execute 8 sec latency 9.600 ms oleg254-client.virtnet: 1 21139 3.74 MB/sec execute 9 sec latency 10.194 ms oleg254-client.virtnet: 1 21797 3.83 MB/sec execute 10 sec latency 8.201 ms oleg254-client.virtnet: 1 22329 3.57 MB/sec execute 11 sec latency 18.127 ms oleg254-client.virtnet: 1 22890 3.62 MB/sec execute 12 sec latency 12.904 ms oleg254-client.virtnet: 1 23455 3.39 MB/sec execute 13 sec latency 10.192 ms oleg254-client.virtnet: 1 24249 3.63 MB/sec execute 14 sec latency 10.561 ms oleg254-client.virtnet: 1 25131 3.81 MB/sec execute 15 sec latency 10.410 ms oleg254-client.virtnet: 1 25621 3.61 MB/sec execute 16 sec latency 15.642 ms oleg254-client.virtnet: 1 26105 3.50 MB/sec execute 17 sec latency 15.212 ms oleg254-client.virtnet: 1 26612 3.49 MB/sec execute 18 sec latency 13.943 ms oleg254-client.virtnet: 1 27226 3.39 MB/sec execute 19 sec lateUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594156 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3568760 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162916 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:12:57 (1734433977) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:13:11 (1734433991) targets are mounted 06:13:11 (1734433991) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568660 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7162716 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:13:23 (1734434003) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 ncy 11.420 ms oleg254-client.virtnet: 1 28080 3.54 MB/sec execute 20 sec latency 11.838 ms oleg254-client.virtnet: 1 28679 3.63 MB/sec execute 21 sec latency 15.269 ms oleg254-client.virtnet: 1 28984 3.48 MB/sec execute 22 sec latency 14.910 ms oleg254-client.virtnet: 1 29396 3.37 MB/sec execute 23 sec latency 16.870 ms oleg254-client.virtnet: 1 29700 3.28 MB/sec execute 24 sec latency 437.938 ms oleg254-client.virtnet: 1 29700 3.15 MB/sec execute 25 sec latency 1438.087 ms oleg254-client.virtnet: 1 29700 3.03 MB/sec execute 26 sec latency 2438.309 ms oleg254-client.virtnet: 1 29700 2.92 MB/sec execute 27 sec latency 3438.423 ms oleg254-client.virtnet: 1 29700 2.81 MB/sec execute 28 sec latency 4438.565 ms oleg254-client.virtnet: 1 29700 2.71 MB/sec execute 29 sec latency 5438.735 ms oleg254-client.virtnet: 1 29700 2.62 MB/sec execute 30 sec latency 6438.878 ms oleg254-client.virtnet: 1 29700 2.54 MB/sec execute 31 sec latency 7439.019 ms oleg254-client.virtnet: 1 29700 2.46 MB/sec execute 32 sec latency 8439.174 ms oleg254-client.virtnet: 1 29700 2.39 MB/sec execute 33 sec latency 9439.322 ms oleg254-client.virtnet: 1 29700 2.32 MB/sec execute 34 sec latency 10439.859 ms oleg254-client.virtnet: 1 29700 2.25 MB/sec execute 35 sec latency 11440.038 ms oleg254-client.virtnet: 1 29700 2.19 MB/sec execute 36 sec latency 12440.292 ms oleg254-client.virtnet: 1 29700 2.13 MB/sec execute 37 sec latency 13440.513 ms oleg254-client.virtnet: 1 29700 2.07 MB/sec execute 38 sec latency 14440.685 ms oleg254-client.virtnet: 1 29700 2.02 MB/sec execute 39 sec latency 15440.803 ms oleg254-client.virtnet: 1 29700 1.97 MB/sec execute 40 sec latency 16440.960 ms oleg254-client.virtnet: 1 29700 1.92 MB/sec execute 41 sec latency 17441.150 ms oleg254-client.virtnet: 1 29700 1.87 MB/sec execute 42 sec latency 18441.313 ms oleg254-client.virtnet: 1 29936 1.90 MB/sec execute 43 sec latency 19029.364 ms oleg254-client.virtnet: 1 30364 1.86 MB/sec execute 44 sec latency 17.281 ms oleg254-client.virtnet: 1 30915 1.87 MB/sec execute 45 sec latency 12.019 ms oleg254-client.virtnet: 1 31779 1.98 MB/sec execute 46 sec latency 14.606 ms oleg254-client.virtnet: 1 32290 2.03 MB/sec execute 47 sec latency 10.826 ms oleg254-client.virtnet: 1 32686 2.00 MB/sec execute 48 sec latency 15.482 ms oleg254-client.virtnet: 1 33128 1.99 MB/sec execute 49 sec latency 12.903 ms oleg254-client.virtnet: 1 33595 2.02 MB/sec execute 50 sec latency 14.005 ms oleg254-client.virtnet: 1 33974 1.98 MB/sec execute 51 sec latency 17.796 ms oleg254-client.virtnet: 1 34406 1.98 MB/sec execute 52 sec latency 15.149 ms oleg254-client.virtnet: 1 35347 2.08 MB/sec execute 53 sec latency 13.268 ms oleg254-client.virtnet: 1 35933 2.13 MB/sec execute 54 sec latency 9.910 ms oleg254-client.virtnet: 1 36392 2.10 MB/sec execute 55 sec latency 12.519 ms oleg254-client.virtnet: 1 36962 2.14 MB/sec execute 56 sec latency 11.906 ms oleg254-client.virtnet: 1 37464 2.11 MB/sec execute 57 sec latency 12.187 ms oleg254-client.virtnet: 1 38081 2.12 MB/sec execute 58 sec latency 10.074 ms oleg254-client.virtnet: 1 39173 2.25 MB/sec execute 59 sec latency 8.898 ms oleg254-client.virtnet: 1 39670 2.23 MB/sec execute 60 sec latency 9.772 ms oleg254-client.virtnet: 1 40206 2.23 MB/sec execute 61 sec latency 11.574 ms oleg254-client.virtnet: 1 40777 2.24 MB/sec execute 62 sec latency 15.662 ms oleg254-client.virtnet: 1 41401 2.23 MB/sec execute 63 sec latency 9.504 ms oleg254-client.virtnet: 1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:13:38 (1734434018) targets are mounted 06:13:38 (1734434018) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12080 3593004 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3568816 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7161820 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:13:49 (1734434029) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:14:04 (1734434044) targets are mounted 06:14:04 (1734434044) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12172 3591964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7160784 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:14:16 (1734434056) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server 42042 2.29 MB/sec execute 64 sec latency 19.143 ms oleg254-client.virtnet: 1 42739 2.33 MB/sec execute 65 sec latency 14.807 ms oleg254-client.virtnet: 1 43063 2.32 MB/sec execute 66 sec latency 14.754 ms oleg254-client.virtnet: 1 43471 2.29 MB/sec execute 67 sec latency 18.160 ms oleg254-client.virtnet: 1 43885 2.28 MB/sec execute 68 sec latency 14.112 ms oleg254-client.virtnet: 1 44365 2.29 MB/sec execute 69 sec latency 13.849 ms oleg254-client.virtnet: 1 44758 2.27 MB/sec execute 70 sec latency 14.574 ms oleg254-client.virtnet: 1 45364 2.31 MB/sec execute 71 sec latency 12.916 ms oleg254-client.virtnet: 1 46175 2.37 MB/sec execute 72 sec latency 12.851 ms oleg254-client.virtnet: 1 46570 2.36 MB/sec execute 73 sec latency 13.995 ms oleg254-client.virtnet: 1 46880 2.33 MB/sec execute 74 sec latency 290.927 ms oleg254-client.virtnet: 1 47283 2.32 MB/sec execute 75 sec latency 401.669 ms oleg254-client.virtnet: 1 47804 2.33 MB/sec execute 76 sec latency 12.186 ms oleg254-client.virtnet: 1 47946 2.31 MB/sec execute 77 sec latency 707.076 ms oleg254-client.virtnet: 1 47946 2.28 MB/sec execute 78 sec latency 1707.212 ms oleg254-client.virtnet: 1 47946 2.25 MB/sec execute 79 sec latency 2707.408 ms oleg254-client.virtnet: 1 47946 2.22 MB/sec execute 80 sec latency 3707.599 ms oleg254-client.virtnet: 1 47946 2.19 MB/sec execute 81 sec latency 4707.764 ms oleg254-client.virtnet: 1 47946 2.16 MB/sec execute 82 sec latency 5708.009 ms oleg254-client.virtnet: 1 47946 2.14 MB/sec execute 83 sec latency 6708.174 ms oleg254-client.virtnet: 1 47946 2.11 MB/sec execute 84 sec latency 7708.417 ms oleg254-client.virtnet: 1 47946 2.09 MB/sec execute 85 sec latency 8708.599 ms oleg254-client.virtnet: 1 47946 2.06 MB/sec execute 86 sec latency 9708.780 ms oleg254-client.virtnet: 1 47946 2.04 MB/sec execute 87 sec latency 10708.997 ms oleg254-client.virtnet: 1 47946 2.02 MB/sec execute 88 sec latency 11709.110 ms oleg254-client.virtnet: 1 47946 1.99 MB/sec execute 89 sec latency 12709.280 ms oleg254-client.virtnet: 1 47946 1.97 MB/sec execute 90 sec latency 13709.418 ms oleg254-client.virtnet: 1 47946 1.95 MB/sec execute 91 sec latency 14709.489 ms oleg254-client.virtnet: 1 47946 1.93 MB/sec execute 92 sec latency 15709.644 ms oleg254-client.virtnet: 1 47946 1.91 MB/sec execute 93 sec latency 16709.803 ms oleg254-client.virtnet: 1 47946 1.89 MB/sec execute 94 sec latency 17709.997 ms oleg254-client.virtnet: 1 48010 1.87 MB/sec execute 95 sec latency 18552.058 ms oleg254-client.virtnet: 1 48597 1.87 MB/sec execute 96 sec latency 11.444 ms oleg254-client.virtnet: 1 49634 1.95 MB/sec execute 97 sec latency 14.417 ms oleg254-client.virtnet: 1 50079 1.95 MB/sec execute 98 sec latency 11.681 ms oleg254-client.virtnet: 1 50440 1.94 MB/sec execute 99 sec latency 25.802 ms oleg254-client.virtnet: 1 50846 1.93 MB/sec execute 100 sec latency 14.223 ms oleg254-client.virtnet: 1 51385 1.95 MB/sec execute 101 sec latency 12.646 ms oleg254-client.virtnet: 1 51860 1.93 MB/sec execute 102 sec latency 11.643 ms oleg254-client.virtnet: 1 52586 1.97 MB/sec execute 103 sec latency 14.949 ms oleg254-client.virtnet: 1 53407 2.01 MB/sec execute 104 sec latency 11.977 ms oleg254-client.virtnet: 1 53954 2.01 MB/sec execute 105 sec latency 12.027 ms oleg254-client.virtnet: 1 54621 2.03 MB/sec execute 106 sec latency 8.796 ms oleg254-client.virtnet: 1 55147 2.02 MB/sec execute 107 sec latency 9.276 ms oleg254-client.virtnet: 1 55814 2.02 MB/sec execute 108 sec latencFailover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:14:30 (1734434070) targets are mounted 06:14:30 (1734434070) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12084 3591884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37552 3568880 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160764 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:14:42 (1734434082) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:14:56 (1734434096) targets are mounted 06:14:56 (1734434096) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12220 3591364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37604 3568856 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49824 7160220 1% /mnt/lustre y 8.528 ms oleg254-client.virtnet: 1 56904 2.09 MB/sec execute 109 sec latency 10.731 ms oleg254-client.virtnet: 1 57334 2.09 MB/sec execute 110 sec latency 11.933 ms oleg254-client.virtnet: 1 57827 2.08 MB/sec execute 111 sec latency 14.941 ms oleg254-client.virtnet: 1 58391 2.10 MB/sec execute 112 sec latency 12.361 ms oleg254-client.virtnet: 1 58796 2.08 MB/sec execute 113 sec latency 19.894 ms oleg254-client.virtnet: 1 59333 2.08 MB/sec execute 114 sec latency 14.148 ms oleg254-client.virtnet: 1 60168 2.12 MB/sec execute 115 sec latency 14.803 ms oleg254-client.virtnet: 1 60696 2.14 MB/sec execute 116 sec latency 14.066 ms oleg254-client.virtnet: 1 61138 2.13 MB/sec execute 117 sec latency 11.690 ms oleg254-client.virtnet: 1 61564 2.12 MB/sec execute 118 sec latency 17.856 ms oleg254-client.virtnet: 1 62060 2.13 MB/sec execute 119 sec latency 12.502 ms oleg254-client.virtnet: 1 62440 2.12 MB/sec execute 120 sec latency 16.609 ms oleg254-client.virtnet: 1 62906 2.12 MB/sec execute 121 sec latency 16.328 ms oleg254-client.virtnet: 1 63756 2.16 MB/sec execute 122 sec latency 14.331 ms oleg254-client.virtnet: 1 64149 2.17 MB/sec execute 123 sec latency 13.681 ms oleg254-client.virtnet: 1 64522 2.16 MB/sec execute 124 sec latency 13.473 ms oleg254-client.virtnet: 1 64980 2.16 MB/sec execute 125 sec latency 13.499 ms oleg254-client.virtnet: 1 65505 2.17 MB/sec execute 126 sec latency 14.082 ms oleg254-client.virtnet: 1 65721 2.15 MB/sec execute 127 sec latency 413.008 ms oleg254-client.virtnet: 1 66284 2.15 MB/sec execute 128 sec latency 14.563 ms oleg254-client.virtnet: 1 66837 2.17 MB/sec execute 129 sec latency 421.754 ms oleg254-client.virtnet: 1 66837 2.16 MB/sec execute 130 sec latency 1421.911 ms oleg254-client.virtnet: 1 66837 2.14 MB/sec execute 131 sec latency 2422.091 ms oleg254-client.virtnet: 1 66837 2.12 MB/sec execute 132 sec latency 3422.275 ms oleg254-client.virtnet: 1 66837 2.11 MB/sec execute 133 sec latency 4422.464 ms oleg254-client.virtnet: 1 66837 2.09 MB/sec execute 134 sec latency 5422.613 ms oleg254-client.virtnet: 1 66837 2.08 MB/sec execute 135 sec latency 6422.748 ms oleg254-client.virtnet: 1 66837 2.06 MB/sec execute 136 sec latency 7422.891 ms oleg254-client.virtnet: 1 66837 2.05 MB/sec execute 137 sec latency 8423.051 ms oleg254-client.virtnet: 1 66837 2.03 MB/sec execute 138 sec latency 9423.211 ms oleg254-client.virtnet: 1 66837 2.02 MB/sec execute 139 sec latency 10423.394 ms oleg254-client.virtnet: 1 66837 2.00 MB/sec execute 140 sec latency 11423.580 ms oleg254-client.virtnet: 1 66837 1.99 MB/sec execute 141 sec latency 12423.733 ms oleg254-client.virtnet: 1 66837 1.97 MB/sec execute 142 sec latency 13423.871 ms oleg254-client.virtnet: 1 66837 1.96 MB/sec execute 143 sec latency 14423.993 ms oleg254-client.virtnet: 1 66837 1.95 MB/sec execute 144 sec latency 15424.096 ms oleg254-client.virtnet: 1 66837 1.93 MB/sec execute 145 sec latency 16424.240 ms oleg254-client.virtnet: 1 66837 1.92 MB/sec execute 146 sec latency 17424.470 ms oleg254-client.virtnet: 1 66842 1.91 MB/sec execute 147 sec latency 18405.709 ms oleg254-client.virtnet: 1 67435 1.93 MB/sec execute 148 sec latency 16.232 ms oleg254-client.virtnet: 1 67863 1.93 MB/sec execute 149 sec latency 10.300 ms oleg254-client.virtnet: 1 68305 1.92 MB/sec execute 150 sec latency 15.208 ms oleg254-client.virtnet: 1 68886 1.93 MB/sec execute 151 sec latency 11.154 ms oleg254-client.virtnet: 1 69412 1.92 MB/sec execute 152 sec latency 11.415 ms oleg254-client.virtnet: 1 6test_70b fail mds1 9 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:15:08 (1734434108) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:22 (1734434122) targets are mounted 06:15:22 (1734434122) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12476 3593528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569016 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49892 7162544 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:15:34 (1734434134) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:15:48 (1734434148) targets are mounted 06:15:48 (1734434148) facet_failover done 9905 1.93 MB/sec execute 153 sec latency 13.052 ms oleg254-client.virtnet: 1 70785 1.96 MB/sec execute 154 sec latency 13.427 ms oleg254-client.virtnet: 1 71291 1.97 MB/sec execute 155 sec latency 19.521 ms oleg254-client.virtnet: 1 71752 1.96 MB/sec execute 156 sec latency 11.206 ms oleg254-client.virtnet: 1 72243 1.96 MB/sec execute 157 sec latency 10.348 ms oleg254-client.virtnet: 1 72802 1.97 MB/sec execute 158 sec latency 11.970 ms oleg254-client.virtnet: 1 73325 1.97 MB/sec execute 159 sec latency 12.241 ms oleg254-client.virtnet: 1 73989 1.99 MB/sec execute 160 sec latency 14.068 ms oleg254-client.virtnet: 1 74713 2.01 MB/sec execute 161 sec latency 11.763 ms oleg254-client.virtnet: 1 75054 2.01 MB/sec execute 162 sec latency 13.419 ms oleg254-client.virtnet: 1 75416 2.00 MB/sec execute 163 sec latency 20.421 ms oleg254-client.virtnet: 1 75775 2.00 MB/sec execute 164 sec latency 14.273 ms oleg254-client.virtnet: 1 76318 2.00 MB/sec execute 165 sec latency 14.105 ms oleg254-client.virtnet: 1 76872 2.00 MB/sec execute 166 sec latency 11.113 ms oleg254-client.virtnet: 1 77554 2.02 MB/sec execute 167 sec latency 11.662 ms oleg254-client.virtnet: 1 78331 2.05 MB/sec execute 168 sec latency 11.567 ms oleg254-client.virtnet: 1 78787 2.04 MB/sec execute 169 sec latency 13.102 ms oleg254-client.virtnet: 1 79194 2.04 MB/sec execute 170 sec latency 14.544 ms oleg254-client.virtnet: 1 79711 2.04 MB/sec execute 171 sec latency 13.069 ms oleg254-client.virtnet: 1 80280 2.04 MB/sec execute 172 sec latency 12.119 ms oleg254-client.virtnet: 1 81290 2.07 MB/sec execute 173 sec latency 9.204 ms oleg254-client.virtnet: 1 81981 2.09 MB/sec execute 174 sec latency 11.546 ms oleg254-client.virtnet: 1 82467 2.08 MB/sec execute 175 sec latency 15.144 ms oleg254-client.virtnet: 1 83223 2.09 MB/sec execute 176 sec latency 7.742 ms oleg254-client.virtnet: 1 83969 2.09 MB/sec execute 177 sec latency 8.276 ms oleg254-client.virtnet: 1 84632 2.11 MB/sec execute 178 sec latency 290.925 ms oleg254-client.virtnet: 1 85469 2.14 MB/sec execute 179 sec latency 421.735 ms oleg254-client.virtnet: 1 85900 2.13 MB/sec execute 180 sec latency 13.520 ms oleg254-client.virtnet: 1 86056 2.12 MB/sec execute 181 sec latency 667.476 ms oleg254-client.virtnet: 1 86056 2.11 MB/sec execute 182 sec latency 1667.658 ms oleg254-client.virtnet: 1 86056 2.09 MB/sec execute 183 sec latency 2667.869 ms oleg254-client.virtnet: 1 86056 2.08 MB/sec execute 184 sec latency 3668.056 ms oleg254-client.virtnet: 1 86056 2.07 MB/sec execute 185 sec latency 4668.252 ms oleg254-client.virtnet: 1 86056 2.06 MB/sec execute 186 sec latency 5668.444 ms oleg254-client.virtnet: 1 86056 2.05 MB/sec execute 187 sec latency 6668.809 ms oleg254-client.virtnet: 1 86056 2.04 MB/sec execute 188 sec latency 7669.026 ms oleg254-client.virtnet: 1 86056 2.03 MB/sec execute 189 sec latency 8669.219 ms oleg254-client.virtnet: 1 86056 2.02 MB/sec execute 190 sec latency 9669.396 ms oleg254-client.virtnet: 1 86056 2.01 MB/sec execute 191 sec latency 10669.546 ms oleg254-client.virtnet: 1 86056 2.00 MB/sec execute 192 sec latency 11670.484 ms oleg254-client.virtnet: 1 86056 1.99 MB/sec execute 193 sec latency 12671.646 ms oleg254-client.virtnet: 1 86056 1.98 MB/sec execute 194 sec latency 13672.695 ms oleg254-client.virtnet: 1 86056 1.96 MB/sec execute 195 sec latency 14674.495 ms oleg254-client.virtnet: 1 86056 1.95 MB/sec execute 196 sec latency 15674.698 ms oleg254-client.virtnet: 1 86056 1.95 MB/sec execute 197 sec latency 16674.847oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12132 3591728 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3568756 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49556 7160484 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:16:00 (1734434160) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:14 (1734434174) targets are mounted 06:16:14 (1734434174) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12476 3593528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569016 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49892 7162544 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:16:26 (1734434186) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server ms oleg254-client.virtnet: 1 86056 1.94 MB/sec execute 198 sec latency 17674.981 ms oleg254-client.virtnet: 1 86056 1.93 MB/sec execute 199 sec latency 18675.103 ms oleg254-client.virtnet: 1 86344 1.92 MB/sec execute 200 sec latency 18865.169 ms oleg254-client.virtnet: 1 86835 1.93 MB/sec execute 201 sec latency 13.353 ms oleg254-client.virtnet: 1 87378 1.92 MB/sec execute 202 sec latency 9.708 ms oleg254-client.virtnet: 1 88344 1.95 MB/sec execute 203 sec latency 10.672 ms oleg254-client.virtnet: 1 89050 1.97 MB/sec execute 204 sec latency 11.303 ms oleg254-client.virtnet: 1 89502 1.96 MB/sec execute 205 sec latency 12.488 ms oleg254-client.virtnet: 1 90029 1.96 MB/sec execute 206 sec latency 12.396 ms oleg254-client.virtnet: 1 90562 1.97 MB/sec execute 207 sec latency 11.391 ms oleg254-client.virtnet: 1 91292 1.97 MB/sec execute 208 sec latency 8.681 ms oleg254-client.virtnet: 1 92415 2.01 MB/sec execute 209 sec latency 8.431 ms oleg254-client.virtnet: 1 92932 2.00 MB/sec execute 210 sec latency 11.464 ms oleg254-client.virtnet: 1 93415 2.00 MB/sec execute 211 sec latency 13.928 ms oleg254-client.virtnet: 1 93986 2.01 MB/sec execute 212 sec latency 13.011 ms oleg254-client.virtnet: 1 94478 2.00 MB/sec execute 213 sec latency 11.867 ms oleg254-client.virtnet: 1 95457 2.03 MB/sec execute 214 sec latency 10.274 ms oleg254-client.virtnet: 1 96202 2.05 MB/sec execute 215 sec latency 8.436 ms oleg254-client.virtnet: 1 96700 2.04 MB/sec execute 216 sec latency 11.749 ms oleg254-client.virtnet: 1 97327 2.05 MB/sec execute 217 sec latency 9.106 ms oleg254-client.virtnet: 1 97907 2.04 MB/sec execute 218 sec latency 9.586 ms oleg254-client.virtnet: 1 98737 2.07 MB/sec execute 219 sec latency 9.369 ms oleg254-client.virtnet: 1 99578 2.08 MB/sec execute 220 sec latency 10.025 ms oleg254-client.virtnet: 1 100083 2.08 MB/sec execute 221 sec latency 12.513 ms oleg254-client.virtnet: 1 100592 2.08 MB/sec execute 222 sec latency 10.118 ms oleg254-client.virtnet: 1 101179 2.08 MB/sec execute 223 sec latency 11.516 ms oleg254-client.virtnet: 1 101829 2.08 MB/sec execute 224 sec latency 8.415 ms oleg254-client.virtnet: 1 102935 2.12 MB/sec execute 225 sec latency 10.245 ms oleg254-client.virtnet: 1 103347 2.12 MB/sec execute 226 sec latency 12.049 ms oleg254-client.virtnet: 1 103784 2.11 MB/sec execute 227 sec latency 14.315 ms oleg254-client.virtnet: 1 104179 2.11 MB/sec execute 228 sec latency 15.907 ms oleg254-client.virtnet: 1 104691 2.11 MB/sec execute 229 sec latency 11.650 ms oleg254-client.virtnet: 1 105159 2.10 MB/sec execute 230 sec latency 12.076 ms oleg254-client.virtnet: 1 105781 2.12 MB/sec execute 231 sec latency 195.745 ms oleg254-client.virtnet: 1 106566 2.14 MB/sec execute 232 sec latency 11.142 ms oleg254-client.virtnet: 1 106759 2.13 MB/sec execute 233 sec latency 466.879 ms oleg254-client.virtnet: 1 106759 2.12 MB/sec execute 234 sec latency 1467.076 ms oleg254-client.virtnet: 1 106759 2.11 MB/sec execute 235 sec latency 2467.260 ms oleg254-client.virtnet: 1 106759 2.11 MB/sec execute 236 sec latency 3467.439 ms oleg254-client.virtnet: 1 106759 2.10 MB/sec execute 237 sec latency 4467.591 ms oleg254-client.virtnet: 1 106759 2.09 MB/sec execute 238 sec latency 5467.792 ms oleg254-client.virtnet: 1 106759 2.08 MB/sec execute 239 sec latency 6468.518 ms oleg254-client.virtnet: 1 106759 2.07 MB/sec execute 240 sec latency 7468.718 ms oleg254-client.virtnet: 1 106759 2.06 MB/sec execute 241 sec latency 8468.894 ms oleg254-client.virtnet: 1 106759 2.05 MB/sec execute 242 sFailover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:16:40 (1734434200) targets are mounted 06:16:40 (1734434200) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ec latency 9469.069 ms oleg254-client.virtnet: 1 106759 2.04 MB/sec execute 243 sec latency 10469.225 ms oleg254-client.virtnet: 1 106759 2.04 MB/sec execute 244 sec latency 11469.321 ms oleg254-client.virtnet: 1 106759 2.03 MB/sec execute 245 sec latency 12469.469 ms oleg254-client.virtnet: 1 106759 2.02 MB/sec execute 246 sec latency 13469.654 ms oleg254-client.virtnet: 1 106759 2.01 MB/sec execute 247 sec latency 14469.844 ms oleg254-client.virtnet: 1 106759 2.00 MB/sec execute 248 sec latency 15470.011 ms oleg254-client.virtnet: 1 106759 2.00 MB/sec execute 249 sec latency 16470.255 ms oleg254-client.virtnet: 1 106759 1.99 MB/sec execute 250 sec latency 17470.411 ms oleg254-client.virtnet: 1 106759 1.98 MB/sec execute 251 sec latency 18470.895 ms oleg254-client.virtnet: 1 106939 1.97 MB/sec execute 252 sec latency 18820.202 ms oleg254-client.virtnet: 1 107221 1.97 MB/sec execute 253 sec latency 28.356 ms oleg254-client.virtnet: 1 107510 1.97 MB/sec execute 254 sec latency 19.959 ms oleg254-client.virtnet: 1 107923 1.97 MB/sec execute 255 sec latency 16.131 ms oleg254-client.virtnet: 1 108277 1.96 MB/sec execute 256 sec latency 13.798 ms oleg254-client.virtnet: 1 108802 1.96 MB/sec execute 257 sec latency 12.447 ms oleg254-client.virtnet: 1 109682 1.98 MB/sec execute 258 sec latency 10.201 ms oleg254-client.virtnet: 1 110321 1.99 MB/sec execute 259 sec latency 11.033 ms oleg254-client.virtnet: 1 110777 1.99 MB/sec execute 260 sec latency 12.848 ms oleg254-client.virtnet: 1 111275 1.99 MB/sec execute 261 sec latency 10.663 ms oleg254-client.virtnet: 1 111822 1.99 MB/sec execute 262 sec latency 11.082 ms oleg254-client.virtnet: 1 112468 1.99 MB/sec execute 263 sec latency 8.779 ms oleg254-client.virtnet: 1 113539 2.02 MB/sec execute 264 sec latency 10.041 ms oleg254-client.virtnet: 1 114030 2.02 MB/sec execute 265 sec latency 9.282 ms oleg254-client.virtnet: 1 114548 2.02 MB/sec execute 266 sec latency 15.853 ms oleg254-client.virtnet: 1 115134 2.02 MB/sec execute 267 sec latency 10.414 ms oleg254-client.virtnet: 1 115604 2.02 MB/sec execute 268 sec latency 15.326 ms oleg254-client.virtnet: 1 116420 2.03 MB/sec execute 269 sec latency 11.763 ms oleg254-client.virtnet: 1 117239 2.05 MB/sec execute 270 sec latency 10.956 ms oleg254-client.virtnet: 1 117714 2.05 MB/sec execute 271 sec latency 9.679 ms oleg254-client.virtnet: 1 118259 2.05 MB/sec execute 272 sec latency 13.698 ms oleg254-client.virtnet: 1 118807 2.05 MB/sec execute 273 sec latency 15.573 ms oleg254-client.virtnet: 1 119241 2.04 MB/sec execute 274 sec latency 12.230 ms oleg254-client.virtnet: 1 119979 2.06 MB/sec execute 275 sec latency 13.331 ms oleg254-client.virtnet: 1 120696 2.07 MB/sec execute 276 sec latency 13.780 ms oleg254-client.virtnet: 1 121029 2.07 MB/sec execute 277 sec latency 17.223 ms oleg254-client.virtnet: 1 121392 2.06 MB/sec execute 278 sec latency 15.156 ms oleg254-client.virtnet: 1 121796 2.06 MB/sec execute 279 sec latency 12.222 ms oleg254-client.virtnet: 1 122284 2.07 MB/sec execute 280 sec latency 12.279 ms oleg254-client.virtnet: 1 122765 2.06 MB/sec execute 281 sec latency 12.502 ms oleg254-client.virtnet: 1 123381 2.07 MB/sec execute 282 sec latency 14.025 ms oleg254-client.virtnet: 1 124190 2.09 MB/sec execute 283 sec latency 15.883 ms oleg254-client.virtnet: 1 124597 2.09 MB/sec execute 284 sec latency 11.952 ms oleg254-client.virtnet: 1 125041 2.08 MB/sec execute 285 sec latency 13.469 ms oleg254-client.virtnet: 1 125454 2.08 MB/sec execute 286 sec latency 12.440 ms oleg254-client.virtnet: 1 125921 2.08 MB/sec execute 287 sec latency 13.668 ms oleg254-client.virtnet: 1 126399 2.08 MB/sec execute 288 sec latency 10.334 ms oleg254-client.virtnet: 1 127163 2.09 MB/sec execute 289 sec latency 15.072 ms oleg254-client.virtnet: 1 127972 2.11 MB/sec execute 290 sec latency 10.964 ms oleg254-client.virtnet: 1 128472 2.10 MB/sec execute 291 sec latency 10.802 ms oleg254-client.virtnet: 1 129132 2.11 MB/sec execute 292 sec latency 10.468 ms oleg254-client.virtnet: 1 129619 2.11 MB/sec execute 293 sec latency 10.787 ms oleg254-client.virtnet: 1 130226 2.11 MB/sec execute 294 sec latency 11.304 ms oleg254-client.virtnet: 1 131257 2.13 MB/sec execute 295 sec latency 11.229 ms oleg254-client.virtnet: 1 131716 2.13 MB/sec execute 296 sec latency 10.972 ms oleg254-client.virtnet: 1 132235 2.13 MB/sec execute 297 sec latency 11.390 ms oleg254-client.virtnet: 1 132795 2.13 MB/sec execute 298 sec latency 12.979 ms oleg254-client.virtnet: 1 133322 2.13 MB/sec execute 299 sec latency 10.303 ms oleg254-client.virtnet: 1 cleanup 300 sec oleg254-client.virtnet: 0 cleanup 300 sec oleg254-client.virtnet: oleg254-client.virtnet: Operation Count AvgLat MaxLat oleg254-client.virtnet: ---------------------------------------- oleg254-client.virtnet: NTCreateX 20637 8.289 18820.193 oleg254-client.virtnet: Close 15184 2.270 18865.149 oleg254-client.virtnet: Rename 873 8.254 20.386 oleg254-client.virtnet: Unlink 4162 6.552 19029.347 oleg254-client.virtnet: Qpathinfo 18742 1.464 16.314 oleg254-client.virtnet: Qfileinfo 3264 0.153 2.827 oleg254-client.virtnet: Qfsinfo 3412 0.223 11.408 oleg254-client.virtnet: Sfileinfo 1698 4.938 18.114 oleg254-client.virtnet: Find 7227 0.410 12.952 oleg254-client.virtnet: WriteX 10201 0.856 9.462 oleg254-client.virtnet: ReadX 32206 0.021 3.932 oleg254-client.virtnet: LockX 66 1.084 2.538 oleg254-client.virtnet: UnlockX 66 0.978 1.838 oleg254-client.virtnet: Flush 1453 6.441 421.722 oleg254-client.virtnet: oleg254-client.virtnet: Throughput 2.1294 MB/sec 1 clients 1 procs max_latency=19029.364 ms oleg254-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Dec 17 06:17:33 EST 2024 with return code 0 oleg254-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg254-client.virtnet: removed directory: 'clients/client0' oleg254-client.virtnet: removed directory: 'clients' oleg254-client.virtnet: removed 'client.txt' oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: dbench successfully finished PASS 70b (363s) == replay-single test 70c: tar 2mdts recovery ============ 06:17:35 (1734434255) Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar 27929 striped dir -i0 -c2 -H crush /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6960 1280728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4628 1283060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 10204 3571008 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6448 3577912 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 16652 7148920 1% /mnt/lustre test_70c fail mds2 1 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:19:49 (1734434389) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:20:03 (1734434403) targets are mounted 06:20:03 (1734434403) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7604 1280084 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4732 1282956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1780 3576652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9964 3569032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11744 7145684 1% /mnt/lustre test_70c fail mds2 2 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:22:22 (1734434542) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:22:37 (1734434557) targets are mounted 06:22:37 (1734434557) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (332s) == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 06:23:06 (1734434586) Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 31199 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6588 1281100 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5096 1282592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3599272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3599236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7198508 1% /mnt/lustre test_70d fail mds1 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:25:20 (1734434720) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:25:34 (1734434734) targets are mounted 06:25:34 (1734434734) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4176 1283512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5272 1282416 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3599272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3599236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7198508 1% /mnt/lustre test_70d fail mds2 2 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:27:55 (1734434875) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:28:09 (1734434889) targets are mounted 06:28:09 (1734434889) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 31199 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (311s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 06:28:17 (1734434897) debug=+ha Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started PID=27529 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4348 1283340 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3920 1283768 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3599272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3599236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7198508 1% /mnt/lustre test_70e fail mds2 1 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:30:31 (1734435031) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:30:44 (1734435044) targets are mounted 06:30:44 (1734435044) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4616 1283072 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3492 1284196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3599272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3599236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7198508 1% /mnt/lustre test_70e fail mds1 2 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:33:04 (1734435184) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:33:18 (1734435198) targets are mounted 06:33:18 (1734435198) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (311s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 06:33:28 (1734435208) mount clients oleg254-client.virtnet ... Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3584700 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3574296 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7158996 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:33:36 (1734435216) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:33:51 (1734435231) targets are mounted 06:33:51 (1734435231) facet_failover done ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2600 3584652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5668 3574296 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 8268 7158948 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:34:04 (1734435244) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... Started lustre-OST0000 06:34:18 (1734435258) targets are mounted 06:34:18 (1734435258) facet_failover done ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3584700 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3549336 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7134036 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 3 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:34:30 (1734435270) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... Started lustre-OST0000 06:34:44 (1734435284) targets are mounted 06:34:44 (1734435284) facet_failover done ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3576388 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3588848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7165236 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 4 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:34:57 (1734435297) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... Started lustre-OST0000 06:35:12 (1734435312) targets are mounted 06:35:12 (1734435312) facet_failover done ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3572196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3555528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7127724 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 5 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:35:25 (1734435325) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Started lustre-OST0000 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... 06:35:39 (1734435339) targets are mounted 06:35:39 (1734435339) facet_failover done ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b382a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b382a800.lru_size=clear PASS 70f (137s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 06:35:45 (1734435345) Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 31418 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3344 1284344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4440 1283248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3564 1284124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2636 1285052 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail mds1 mds2 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:38:02 (1734435482) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 06:38:28 (1734435508) targets are mounted 06:38:28 (1734435508) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2872 1284816 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3708 1283980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3092 1284596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3956 1283732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail mds1 mds2 2 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:40:56 (1734435656) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 06:41:22 (1734435682) targets are mounted 06:41:22 (1734435682) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 31418 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (351s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 06:41:36 (1734435696) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5088 1282600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6092 1281596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:41:40 (1734435700) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:41:53 (1734435713) targets are mounted 06:41:53 (1734435713) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (37s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 06:42:13 (1734435733) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7604 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4000 1283688 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:42:16 (1734435736) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:42:30 (1734435750) targets are mounted 06:42:30 (1734435750) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (40s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 06:42:53 (1734435773) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:42:56 (1734435776) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:43:08 (1734435788) targets are mounted 06:43:08 (1734435788) facet_failover done Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (28s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 06:43:21 (1734435801) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2592 1285096 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:43:24 (1734435804) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:43:38 (1734435818) targets are mounted 06:43:38 (1734435818) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 431.09 ops/second PASS 80a (24s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 06:43:45 (1734435825) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:43:57 (1734435837) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:44:11 (1734435851) targets are mounted 06:44:11 (1734435851) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 440.13 ops/second PASS 80b (33s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 06:44:18 (1734435858) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2668 1285020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4076 1283612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2668 1285020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4076 1283612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:44:23 (1734435863) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:44:37 (1734435877) targets are mounted 06:44:37 (1734435877) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:44:45 (1734435885) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:44:59 (1734435899) targets are mounted 06:44:59 (1734435899) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 432.14 ops/second PASS 80c (48s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 06:45:06 (1734435906) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:45:21 (1734435921) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 06:45:41 (1734435941) targets are mounted 06:45:41 (1734435941) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 389.92 ops/second PASS 80d (42s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 06:45:48 (1734435948) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:45:55 (1734435955) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:46:09 (1734435969) targets are mounted 06:46:09 (1734435969) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 436.58 ops/second PASS 80e (28s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 06:46:16 (1734435976) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:46:20 (1734435980) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:46:34 (1734435994) targets are mounted 06:46:34 (1734435994) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 439.46 ops/second PASS 80f (25s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 06:46:41 (1734436001) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:46:49 (1734436009) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:47:03 (1734436023) targets are mounted 06:47:03 (1734436023) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:47:10 (1734436030) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:47:24 (1734436044) targets are mounted 06:47:24 (1734436044) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 411.94 ops/second PASS 80g (50s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 06:47:31 (1734436051) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:47:41 (1734436061) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 06:48:07 (1734436087) targets are mounted 06:48:07 (1734436087) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 431.26 ops/second PASS 80h (46s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 06:48:17 (1734436097) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4040 1283648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:48:26 (1734436106) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:48:40 (1734436120) targets are mounted 06:48:40 (1734436120) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (30s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 06:48:47 (1734436127) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1285024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4044 1283644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:48:51 (1734436131) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:49:05 (1734436145) targets are mounted 06:49:05 (1734436145) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (25s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 06:49:12 (1734436152) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4044 1283644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4044 1283644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:49:17 (1734436157) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:49:31 (1734436171) targets are mounted 06:49:31 (1734436171) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:49:38 (1734436178) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:49:52 (1734436192) targets are mounted 06:49:52 (1734436192) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (47s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 06:49:59 (1734436199) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1285024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1285024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:50:11 (1734436211) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 06:50:32 (1734436232) targets are mounted 06:50:32 (1734436232) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (52s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 06:50:51 (1734436251) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1285024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4084 1283604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:50:55 (1734436255) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:51:09 (1734436269) targets are mounted 06:51:09 (1734436269) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (25s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 06:51:16 (1734436276) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:51:19 (1734436279) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:51:33 (1734436293) targets are mounted 06:51:33 (1734436293) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (25s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 06:51:41 (1734436301) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:51:46 (1734436306) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:52:00 (1734436320) targets are mounted 06:52:00 (1734436320) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:52:08 (1734436328) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:52:22 (1734436342) targets are mounted 06:52:22 (1734436342) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (49s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 06:52:30 (1734436350) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:52:38 (1734436358) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 06:53:03 (1734436383) targets are mounted 06:53:03 (1734436383) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (42s) == replay-single test 84a: stale open during export disconnect ========================================================== 06:53:12 (1734436392) fail_loc=0x80000144 total: 1 open/close in 0.00 seconds: 207.68 ops/second oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 PASS 84a (5s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 06:53:17 (1734436397) before recovery: unused locks count = 201 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:53:19 (1734436399) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:53:32 (1734436412) targets are mounted 06:53:32 (1734436412) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (23s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 06:53:40 (1734436420) before recovery: unused locks count = 100 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:53:44 (1734436424) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:53:57 (1734436437) targets are mounted 06:53:57 (1734436437) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (22s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 06:54:02 (1734436442) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) PASS 86 (10s) == replay-single test 87a: write replay ================== 06:54:12 (1734436452) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2624 1285064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4052 1283636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1776 3605244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3548 7210492 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.111766 s, 75.1 MB/s Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:54:16 (1734436456) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:54:31 (1734436471) targets are mounted 06:54:31 (1734436471) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0693757 s, 121 MB/s PASS 87a (25s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 06:54:37 (1734436477) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2624 1285064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4052 1283636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9968 3597052 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11740 7202300 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.129358 s, 64.8 MB/s 8+0 records in 8+0 records out 8 bytes (8 B) copied, 0.00191362 s, 4.2 kB/s Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:54:42 (1734436482) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:54:56 (1734436496) targets are mounted 06:54:56 (1734436496) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes (72 B) copied, 0.00526699 s, 13.7 kB/s PASS 87b (24s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 06:55:01 (1734436501) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4052 1283636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9972 3597048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11744 7202296 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4052 1283636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9972 3597048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11744 7202296 1% /mnt/lustre before test: last_id = 17281, next_id = 17252 Creating to objid 17281 on ost lustre-OST0000... total: 31 open/close in 0.07 seconds: 431.77 ops/second total: 8 open/close in 0.02 seconds: 400.62 ops/second before recovery: last_id = 17313, next_id = 17290 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg254-server oleg254-server.virtnet Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 17321, next_id = 17290 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.03536 s, 14.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0327139 s, 16.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0391916 s, 13.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0408792 s, 12.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0411228 s, 12.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0269276 s, 19.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.024368 s, 21.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0225976 s, 23.2 MB/s -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17252 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17253 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17254 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17255 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17256 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17257 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17258 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17259 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17260 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17261 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17262 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17263 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17264 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17265 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17266 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17267 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17268 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17269 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17270 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17271 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17272 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17273 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17274 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17275 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17276 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17277 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17278 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17279 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17280 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17281 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17282 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17283 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17284 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17285 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17286 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17287 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17288 -rw-r--r-- 1 root root 0 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17289 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17293 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17294 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17295 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17296 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17297 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17298 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17299 -rw-r--r-- 1 root root 524288 Dec 17 06:55 /mnt/lustre/d88.replay-single/f-17300 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0318688 s, 16.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0344184 s, 15.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0224864 s, 23.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0225353 s, 23.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.033048 s, 15.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0335065 s, 15.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0347853 s, 15.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0428904 s, 12.2 MB/s PASS 88 (52s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 06:55:53 (1734436553) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0631193 s, 166 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:55:59 (1734436559) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:56:12 (1734436572) targets are mounted 06:56:12 (1734436572) facet_failover done Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 free_before: 7646296 free_after: 7646296 PASS 89 (113s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 06:57:46 (1734436666) Create the files Fail ost1 lustre-OST0000_UUID, display the list of affected files Stopping /mnt/lustre-ost1 (opts:) on oleg254-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Querying files on shutdown ost1: lfs find --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 17322 0x43aa 0x280000401 * /mnt/lustre/d90.replay-single/f0 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 17323 0x43ab 0x280000401 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Failover ost1 to oleg254-server oleg254-server.virtnet Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 90 (18s) == replay-single test 93a: replay + reconnect ============ 06:58:04 (1734436684) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00128233 s, 799 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 06:58:05 (1734436685) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 06:58:18 (1734436698) targets are mounted 06:58:18 (1734436698) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (58s) == replay-single test 93b: replay + reconnect on mds ===== 06:59:02 (1734436742) total: 20 open/close in 0.07 seconds: 290.78 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:59:04 (1734436744) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:59:17 (1734436757) targets are mounted 06:59:17 (1734436757) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (102s) == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 07:00:44 (1734436844) fail_loc=0x1701 Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:00:46 (1734436846) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:00:59 (1734436859) targets are mounted 07:00:59 (1734436859) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200033080:0x1:0x0] 1 [0x24000c36a:0x1:0x0] total: 20 open/close in 0.05 seconds: 412.35 ops/second PASS 100a (23s) == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 07:01:07 (1734436867) fail_loc=0x119 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:01:14 (1734436874) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:01:27 (1734436887) targets are mounted 07:01:27 (1734436887) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200033080:0x4:0x0] 1 [0x24000c36a:0x4:0x0] total: 20 open/close in 0.05 seconds: 408.29 ops/second PASS 100b (27s) == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 07:01:34 (1734436894) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds2 (opts:) on oleg254-server Failover mds2 to oleg254-server oleg254-server.virtnet Starting mds2: -o localrecov -o abort_recov_mdt /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 total: 20 open/close in 0.08 seconds: 242.22 ops/second Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:01:53 (1734436913) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:02:06 (1734436926) targets are mounted 07:02:06 (1734436926) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 1 [0x24000cb38:0x3:0x0] 0 [0x200033081:0x3:0x0] total: 20 open/close in 0.04 seconds: 476.31 ops/second PASS 100c (40s) == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 07:02:14 (1734436934) striped dir -i0 -c2 -H all_char /mnt/lustre/d100d.replay-single total: 100 mkdir in 0.30 seconds: 335.23 ops/second lustre-MDT0000-osd [catalog]: [0x20001a210:0x1:0x0] [index]: 00026 [logid]: [0x200033850:0x1:0x0] lustre-MDT0001-osp-MDT0000 [catalog]: [0x2400007ec:0x1:0x0] [index]: 00035 [logid]: [0x24000cb39:0x1:0x0] [index]: 00036 [logid]: [0x24000cb39:0x2:0x0] Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 find: '/mnt/lustre/d100d.replay-single': No such file or directory find: '/mnt/lustre/d100d.replay-single': No such file or directory PASS 100d (18s) == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 07:02:32 (1734436952) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200034020:0x2:0x0] 1 [0x24000dad9:0x2:0x0] Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:02:42 (1734436962) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 07:03:07 (1734436987) targets are mounted 07:03:07 (1734436987) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200034020:0x2:0x0] 1 [0x24000dad9:0x2:0x0] PASS 100e (45s) == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 07:03:17 (1734436997) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3056 1284632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2600 1285088 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 first stat failed: 5 PASS 101 (48s) == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 07:04:05 (1734437045) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (07:04:07) ... fail_loc=0 done (07:04:23) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (19s) == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 07:04:24 (1734437064) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (07:04:25) ... fail_loc=0 done (07:04:41) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (19s) == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 07:04:43 (1734437083) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2972 1284716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3580648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3597044 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7177692 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (07:04:47) ... fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:04:50 (1734437090) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:05:05 (1734437105) targets are mounted 07:05:05 (1734437105) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (07:05:11) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (30s) == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 07:05:13 (1734437113) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (07:05:14) ... fail_loc=0 Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:05:16 (1734437116) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:05:29 (1734437129) targets are mounted 07:05:29 (1734437129) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec done (07:05:35) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (24s) == replay-single test 103: Check otr_next_id overflow ==== 07:05:37 (1734437137) fail_loc=0x80000162 total: 30 open/close in 0.15 seconds: 203.90 ops/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:05:39 (1734437139) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:05:52 (1734437152) targets are mounted 07:05:52 (1734437152) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (22s) == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 07:05:59 (1734437159) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2920 1284768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3580648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3597044 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7177692 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:06:03 (1734437163) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:06:17 (1734437177) targets are mounted 07:06:17 (1734437177) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110a.replay-single/striped_dir has type dir OK PASS 110a (25s) == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 07:06:24 (1734437184) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2924 1284764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3580648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7185896 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:06:28 (1734437188) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:06:42 (1734437202) targets are mounted 07:06:42 (1734437202) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre /mnt/lustre/d110b.replay-single/striped_dir has type dir OK PASS 110b (94s) == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 07:07:58 (1734437278) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2928 1284760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:08:01 (1734437281) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:08:15 (1734437295) targets are mounted 07:08:15 (1734437295) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110c.replay-single/striped_dir has type dir OK PASS 110c (24s) == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 07:08:22 (1734437302) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2968 1284720 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:08:26 (1734437306) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:08:39 (1734437319) targets are mounted 07:08:39 (1734437319) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre /mnt/lustre/d110d.replay-single/striped_dir has type dir OK PASS 110d (92s) == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 07:09:54 (1734437394) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3008 1284680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:10:01 (1734437401) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 07:10:27 (1734437427) targets are mounted 07:10:27 (1734437427) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre /mnt/lustre/d110e.replay-single/striped_dir has type dir OK PASS 110e (108s) SKIP: replay-single test_110f skipping excluded test 110f == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 07:11:42 (1734437502) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3048 1284640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:11:54 (1734437514) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 07:12:15 (1734437535) targets are mounted 07:12:15 (1734437535) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre /mnt/lustre/d110g.replay-single/striped_dir has type dir OK PASS 110g (104s) == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 07:13:26 (1734437606) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2932 1284756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:13:30 (1734437610) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:13:44 (1734437624) targets are mounted 07:13:44 (1734437624) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111a.replay-single/striped_dir: No such file or directory PASS 111a (25s) == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 07:13:51 (1734437631) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2924 1284764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:13:55 (1734437635) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:14:08 (1734437648) targets are mounted 07:14:08 (1734437648) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111b.replay-single/striped_dir: No such file or directory PASS 111b (92s) == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 07:15:23 (1734437723) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2932 1284756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:15:35 (1734437735) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 07:15:55 (1734437755) targets are mounted 07:15:55 (1734437755) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111c.replay-single/striped_dir: No such file or directory PASS 111c (108s) == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 07:17:11 (1734437831) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2940 1284748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:17:17 (1734437837) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 07:17:43 (1734437863) targets are mounted 07:17:43 (1734437863) facet_failover done pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg254-client: oleg254-client: ssh exited with exit code 95 Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111d.replay-single/striped_dir: No such file or directory PASS 111d (107s) == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 07:18:58 (1734437938) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2936 1284752 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:19:05 (1734437945) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 07:19:31 (1734437971) targets are mounted 07:19:31 (1734437971) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111e.replay-single/striped_dir: No such file or directory PASS 111e (41s) == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 07:19:39 (1734437979) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2944 1284744 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2944 1284744 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:19:46 (1734437986) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 07:20:12 (1734438012) targets are mounted 07:20:12 (1734438012) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111f.replay-single/striped_dir: No such file or directory PASS 111f (42s) == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 07:20:21 (1734438021) UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 549 1023451 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 316 1023684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 555 261589 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 489 261655 1% /mnt/lustre[OST:1] filesystem_summary: 524109 865 523244 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2940 1284748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2952 1284736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:20:27 (1734438027) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 07:20:53 (1734438053) targets are mounted 07:20:53 (1734438053) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111g.replay-single/striped_dir: No such file or directory PASS 111g (41s) == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 07:21:02 (1734438062) SKIP: replay-single test_112a needs >= 4 MDTs SKIP 112a (1s) == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 07:21:03 (1734438063) SKIP: replay-single test_112b needs >= 4 MDTs SKIP 112b (0s) == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 07:21:04 (1734438064) SKIP: replay-single test_112c needs >= 4 MDTs SKIP 112c (1s) == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 07:21:04 (1734438064) SKIP: replay-single test_112d needs >= 4 MDTs SKIP 112d (1s) == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 07:21:05 (1734438065) SKIP: replay-single test_112e needs >= 4 MDTs SKIP 112e (1s) == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 07:21:06 (1734438066) SKIP: replay-single test_112f needs >= 4 MDTs SKIP 112f (1s) == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 07:21:07 (1734438067) SKIP: replay-single test_112g needs >= 4 MDTs SKIP 112g (1s) == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 07:21:08 (1734438068) SKIP: replay-single test_112h needs >= 4 MDTs SKIP 112h (1s) == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 07:21:09 (1734438069) SKIP: replay-single test_112i needs >= 4 MDTs SKIP 112i (0s) == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 07:21:09 (1734438069) SKIP: replay-single test_112j needs >= 4 MDTs SKIP 112j (1s) == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 07:21:10 (1734438070) SKIP: replay-single test_112k needs >= 4 MDTs SKIP 112k (1s) == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 07:21:11 (1734438071) SKIP: replay-single test_112l needs >= 4 MDTs SKIP 112l (1s) == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 07:21:12 (1734438072) SKIP: replay-single test_112m needs >= 4 MDTs SKIP 112m (1s) == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 07:21:13 (1734438073) SKIP: replay-single test_112n needs >= 4 MDTs SKIP 112n (0s) == replay-single test 115: failover for create/unlink striped directory ========================================================== 07:21:13 (1734438073) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2948 1284740 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_0 striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_2 striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_3 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_4 Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:21:18 (1734438078) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:21:31 (1734438091) targets are mounted 07:21:31 (1734438091) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3000 1284688 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_0 striped dir -i0 -c2 -H all_char /mnt/lustre/d115.replay-single/test_1 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_2 striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_3 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_4 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:21:41 (1734438101) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:21:55 (1734438115) targets are mounted 07:21:55 (1734438115) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 115 (49s) == replay-single test 116a: large update log master MDT recovery ========================================================== 07:22:02 (1734438122) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3040 1284648 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2584 1285104 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre fail_loc=0x80001702 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:22:06 (1734438126) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:22:21 (1734438141) targets are mounted 07:22:21 (1734438141) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116a.replay-single/striped_dir has type dir OK PASS 116a (26s) == replay-single test 116b: large update log slave MDT recovery ========================================================== 07:22:28 (1734438148) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3092 1284596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2756 1284932 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre fail_loc=0x80001702 Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:22:32 (1734438152) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:22:46 (1734438166) targets are mounted 07:22:46 (1734438166) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116b.replay-single/striped_dir has type dir OK PASS 116b (25s) == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 07:22:53 (1734438173) SKIP: replay-single test_117 needs >= 4 MDTs SKIP 117 (1s) == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 07:22:54 (1734438174) fail_loc=0x1705 lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error fail_loc=0x0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3188 1284500 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2636 1285052 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:22:58 (1734438178) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:23:11 (1734438191) targets are mounted 07:23:11 (1734438191) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 118 (25s) == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 07:23:19 (1734438199) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3256 1284432 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2696 1284992 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet fail_loc=0x80000714 fail_val=65 Starting mds1: -o localrecov -o recovery_time_hard=60 /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 72 sec mdt.lustre-MDT0000.recovery_time_hard=180 PASS 119 (86s) == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 07:24:45 (1734438285) Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 find: '/mnt/lustre/d120.replay-single': No such file or directory find: '/mnt/lustre/d120.replay-single': No such file or directory PASS 120 (21s) == replay-single test 121: lock replay timed out and race ========================================================== 07:25:06 (1734438306) multiop /mnt/lustre/f121.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7604 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server oleg254-server.virtnet fail_loc=0x721 fail_val=0 at_max=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 fail_loc=0x0 at_max=600 PASS 121 (194s) == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 07:28:20 (1734438500) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3648 1284040 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2996 1284692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:28:24 (1734438504) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:28:39 (1734438519) targets are mounted 07:28:39 (1734438519) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130a (27s) == replay-single test 130b: DoM file create (inherited) replay ========================================================== 07:28:47 (1734438527) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3652 1284036 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2996 1284692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:28:51 (1734438531) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:06 (1734438546) targets are mounted 07:29:06 (1734438546) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130b (26s) == replay-single test 131a: DoM file write lock replay === 07:29:13 (1734438553) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3652 1284036 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2996 1284692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre 1+0 records in 1+0 records out 8 bytes (8 B) copied, 0.00135842 s, 5.9 kB/s Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:29:17 (1734438557) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:32 (1734438572) targets are mounted 07:29:32 (1734438572) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 131a (26s) SKIP: replay-single test_131b skipping excluded test 131b == replay-single test 132a: PFL new component instantiate replay ========================================================== 07:29:40 (1734438580) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3664 1284024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2996 1284692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305968 s, 34.3 MB/s /mnt/lustre/f132a.replay-single lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4aea:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4ac2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x4aeb:0x0] } Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:29:44 (1734438584) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:59 (1734438599) targets are mounted 07:29:59 (1734438599) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f132a.replay-single lcm_layout_gen: 4 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4aea:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4ac2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x4aeb:0x0] } PASS 132a (27s) == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 07:30:07 (1734438607) seq.srv-lustre-MDT0001.space=clear Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre fail_val=700 fail_loc=0x80000123 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:30:11 (1734438611) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:30:25 (1734438625) targets are mounted 07:30:25 (1734438625) facet_failover done PASS 133 (22s) == replay-single test 134: replay creation of a file created in a pool ========================================================== 07:30:30 (1734438630) Creating new pool pool_134 oleg254-server: Pool lustre.pool_134 created Adding targets to pool oleg254-server: OST lustre-OST0001_UUID added to pool lustre.pool_134 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3744 1283944 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3040 1284648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2796 3604224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20964 7193076 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:30:38 (1734438638) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:30:53 (1734438653) targets are mounted 07:30:53 (1734438653) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: pool_134 lustre.pool_134 oleg254-server: OST lustre-OST0001_UUID removed from pool lustre.pool_134 oleg254-server: Pool lustre.pool_134 destroyed PASS 134 (37s) == replay-single test 135: Server failure in lock replay phase ========================================================== 07:31:06 (1734438666) Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server 07:31:08 (1734438668) shut down facet: ost1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 07:31:22 (1734438682) targets are mounted 07:31:22 (1734438682) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3724 1283964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3040 1284648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2796 3604224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20964 7193076 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failover ost1 to oleg254-server oleg254-server.virtnet oleg254-server: oleg254-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failover ost1 to oleg254-server oleg254-server.virtnet oleg254-server: oleg254-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 End of sync oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135 (74s) == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 07:32:20 (1734438740) SKIP: replay-single test_136 needs > 2 MDTs SKIP 136 (1s) == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 07:32:21 (1734438741) llite.lustre-ffff8800a7af6000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3732 1283956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3040 1284648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18244 3588776 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 21044 7192996 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:32:25 (1734438745) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:32:38 (1734438758) targets are mounted 07:32:38 (1734438758) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137a.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137a.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137a (25s) == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 07:32:46 (1734438766) llite.lustre-ffff8800a7af6000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3740 1283948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3064 1284624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 07:32:49 (1734438769) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 07:33:03 (1734438783) targets are mounted 07:33:03 (1734438783) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137b.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137b.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137b (24s) == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 07:33:10 (1734438790) llite.lustre-ffff8800a7af6000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3736 1283952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3052 1284636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3736 1283952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3052 1284636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:33:17 (1734438797) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server Failover mds2 to oleg254-server mount facets: mds2 mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 07:33:31 (1734438811) targets are mounted 07:33:31 (1734438811) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137c.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137c.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137c (31s) == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 07:33:41 (1734438821) SKIP: replay-single test_200 Need remote client SKIP 200 (1s) == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 07:33:42 (1734438822) fail_loc=0x245 fail_val=8 fail_loc=0x245 fail_val=8 Stopping /mnt/lustre-mds2 (opts:) on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 Umount took 17 seconds PASS 201 (23s) == replay-single test 202: pfl replay should recovery layout generation ========================================================== 07:34:06 (1734438846) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3688 1284000 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3004 1284684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18168 3588852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 07:34:11 (1734438851) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 07:34:25 (1734438865) targets are mounted 07:34:25 (1734438865) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 202 (28s) == replay-single test complete, duration 8576 sec ======== 07:34:33 (1734438873) === replay-single: start cleanup 07:34:34 (1734438874) === === replay-single: finish cleanup 07:34:39 (1734438879) ===