-----============= acceptance-small: replay-single ============----- Wed May 7 09:45:54 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg148-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 110f 131b 59 36 === replay-single: start setup 09:45:59 (1746625559) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a5de800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5de800.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 09:46:07 (1746625567) === == replay-single test 0a: empty replay =================== 09:46:08 (1746625568) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3072 7210968 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:46:11 (1746625571) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:46:25 (1746625585) targets are mounted 09:46:25 (1746625585) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (24s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 09:46:32 (1746625592) Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 09:46:33 (1746625593) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 09:46:47 (1746625607) targets are mounted 09:46:47 (1746625607) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.07 seconds: 287.40 ops/second - unlinked 0 (time 1746625610 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (19s) == replay-single test 0c: check replay-barrier =========== 09:46:51 (1746625611) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:46:55 (1746625615) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:47:08 (1746625628) targets are mounted 09:47:08 (1746625628) facet_failover done Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (90s) == replay-single test 0d: expired recovery with no clients ========================================================== 09:48:21 (1746625701) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:48:26 (1746625706) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:48:42 (1746625722) targets are mounted 09:48:42 (1746625722) facet_failover done Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 0d (93s) == replay-single test 1: simple create =================== 09:49:54 (1746625794) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:49:58 (1746625798) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:50:12 (1746625812) targets are mounted 09:50:12 (1746625812) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (26s) == replay-single test 2a: touch ========================== 09:50:20 (1746625820) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:50:23 (1746625823) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:50:37 (1746625837) targets are mounted 09:50:37 (1746625837) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (24s) == replay-single test 2b: touch ========================== 09:50:44 (1746625844) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:50:48 (1746625848) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:02 (1746625862) targets are mounted 09:51:02 (1746625862) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (25s) == replay-single test 2c: setstripe replay =============== 09:51:09 (1746625869) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:51:12 (1746625872) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:26 (1746625886) targets are mounted 09:51:26 (1746625886) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (24s) == replay-single test 2d: setdirstripe replay ============ 09:51:33 (1746625893) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1624 1286064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:51:36 (1746625896) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:50 (1746625910) targets are mounted 09:51:50 (1746625910) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (24s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 09:51:57 (1746625917) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:52:02 (1746625922) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:52:16 (1746625936) targets are mounted 09:52:16 (1746625936) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (26s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 09:52:23 (1746625943) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:52:26 (1746625946) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:52:40 (1746625960) targets are mounted 09:52:40 (1746625960) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (25s) == replay-single test 3b: replay failed open -ENOMEM ===== 09:52:48 (1746625968) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:52:51 (1746625971) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:53:05 (1746625985) targets are mounted 09:53:05 (1746625985) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (25s) == replay-single test 3c: replay failed open -ENOMEM ===== 09:53:13 (1746625993) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:53:16 (1746625996) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:53:30 (1746626010) targets are mounted 09:53:30 (1746626010) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (24s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 09:53:37 (1746626017) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210964 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:53:40 (1746626020) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:53:54 (1746626034) targets are mounted 09:53:54 (1746626034) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (24s) == replay-single test 4b: |x| rm 10 files ================ 09:54:01 (1746626041) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605340 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605344 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3076 7210684 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:54:05 (1746626045) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:54:18 (1746626058) targets are mounted 09:54:18 (1746626058) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (25s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 09:54:26 (1746626066) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1285748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:54:31 (1746626071) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:54:46 (1746626086) targets are mounted 09:54:46 (1746626086) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (34s) == replay-single test 6a: mkdir + contained create ======= 09:55:00 (1746626100) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:55:04 (1746626104) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:55:19 (1746626119) targets are mounted 09:55:19 (1746626119) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (28s) == replay-single test 6b: |X| rmdir ====================== 09:55:28 (1746626128) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1968 1285720 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:55:31 (1746626131) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:55:45 (1746626145) targets are mounted 09:55:45 (1746626145) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (25s) == replay-single test 7: mkdir |X| contained create ====== 09:55:53 (1746626153) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1968 1285720 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:55:57 (1746626157) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:56:11 (1746626171) targets are mounted 09:56:11 (1746626171) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (26s) == replay-single test 8: creat open |X| close ============ 09:56:19 (1746626179) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:56:23 (1746626183) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:56:37 (1746626197) targets are mounted 09:56:37 (1746626197) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (25s) == replay-single test 9: |X| create (same inum/gen) ====== 09:56:45 (1746626205) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:56:48 (1746626208) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:57:02 (1746626222) targets are mounted 09:57:02 (1746626222) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (25s) == replay-single test 10: create |X| rename unlink ======= 09:57:10 (1746626230) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:57:13 (1746626233) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:57:27 (1746626247) targets are mounted 09:57:27 (1746626247) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (24s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 09:57:34 (1746626254) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1556 3605464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre new old Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:57:38 (1746626258) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:57:52 (1746626272) targets are mounted 09:57:52 (1746626272) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (25s) == replay-single test 12: open, unlink |X| close ========= 09:57:59 (1746626279) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:58:03 (1746626283) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:58:17 (1746626297) targets are mounted 09:58:17 (1746626297) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (25s) == replay-single test 13: open chmod 0 |x| write close === 09:58:24 (1746626304) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:58:27 (1746626307) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:58:42 (1746626322) targets are mounted 09:58:42 (1746626322) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (25s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 09:58:49 (1746626329) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:58:53 (1746626333) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:59:07 (1746626347) targets are mounted 09:59:07 (1746626347) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (25s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 09:59:14 (1746626354) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:59:18 (1746626358) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:59:32 (1746626372) targets are mounted 09:59:32 (1746626372) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (25s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 09:59:39 (1746626379) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 09:59:43 (1746626383) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 09:59:56 (1746626396) targets are mounted 09:59:56 (1746626396) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (25s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 10:00:04 (1746626404) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:00:07 (1746626407) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:00:21 (1746626421) targets are mounted 10:00:21 (1746626421) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (24s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 10:00:28 (1746626428) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 pid: 22970 will close Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:00:31 (1746626431) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:00:44 (1746626444) targets are mounted 10:00:44 (1746626444) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (23s) == replay-single test 19: mcreate, open, write, rename === 10:00:51 (1746626451) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre old Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:00:55 (1746626455) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:01:08 (1746626468) targets are mounted 10:01:08 (1746626468) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (24s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:01:15 (1746626475) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210912 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:01:18 (1746626478) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:01:32 (1746626492) targets are mounted 10:01:32 (1746626492) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (25s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 10:01:40 (1746626500) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x280000401 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 9930+0 records in 9929+0 records out 40669184 bytes (41 MB) copied, 1.48895 s, 27.3 MB/s Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:01:44 (1746626504) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:01:57 (1746626517) targets are mounted 10:01:57 (1746626517) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg148-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 3128, after 3128 PASS 20b (26s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 10:02:06 (1746626526) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 -rw-r--r-- 1 root root 1 May 7 10:02 /mnt/lustre/f20c.replay-single pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 PASS 20c (4s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:02:10 (1746626530) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210872 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:02:13 (1746626533) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:02:27 (1746626547) targets are mounted 10:02:27 (1746626547) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (25s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:02:35 (1746626555) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210872 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:02:38 (1746626558) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:02:52 (1746626572) targets are mounted 10:02:52 (1746626572) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (24s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:02:59 (1746626579) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:03:03 (1746626583) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:03:16 (1746626596) targets are mounted 10:03:16 (1746626596) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (25s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 10:03:24 (1746626604) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:03:27 (1746626607) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:03:41 (1746626621) targets are mounted 10:03:41 (1746626621) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (24s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:03:48 (1746626628) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:03:52 (1746626632) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:06 (1746626646) targets are mounted 10:04:06 (1746626646) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (25s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:04:13 (1746626653) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:04:16 (1746626656) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:30 (1746626670) targets are mounted 10:04:30 (1746626670) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (25s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:04:38 (1746626678) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:04:41 (1746626681) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:57 (1746626697) targets are mounted 10:04:57 (1746626697) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (26s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:05:04 (1746626704) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:05:08 (1746626708) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:05:22 (1746626722) targets are mounted 10:05:22 (1746626722) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (26s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:05:30 (1746626730) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:05:34 (1746626734) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:05:48 (1746626748) targets are mounted 10:05:48 (1746626748) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (25s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:05:55 (1746626755) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:05:59 (1746626759) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:06:13 (1746626773) targets are mounted 10:06:13 (1746626773) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (26s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 10:06:21 (1746626781) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1285988 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:06:26 (1746626786) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:06:40 (1746626800) targets are mounted 10:06:40 (1746626800) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (26s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 10:06:47 (1746626807) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 PASS 32 (3s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 10:06:50 (1746626810) total: 10 open/close in 0.04 seconds: 249.06 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.04 seconds: 260.78 ops/second PASS 33a (17s) == replay-single test 33b: test fid seq allocation ======= 10:07:07 (1746626827) fail_loc=0x1311 total: 10 open/close in 0.04 seconds: 278.41 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.04 seconds: 226.78 ops/second PASS 33b (17s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 10:07:24 (1746626844) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1285956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 PASS 34 (18s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 10:07:42 (1746626862) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (16s) SKIP: replay-single test_36 skipping ALWAYS excluded test 36 == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 10:07:59 (1746626879) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 PASS 37 (18s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 10:08:17 (1746626897) total: 800 open/close in 2.26 seconds: 353.59 ops/second - unlinked 0 (time 1746626901 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2200 1285488 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:08:25 (1746626905) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:08:39 (1746626919) targets are mounted 10:08:39 (1746626919) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1746626926 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (33s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 10:08:50 (1746626930) total: 800 open/close in 2.29 seconds: 349.96 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2200 1285488 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3132 7210868 1% /mnt/lustre - unlinked 0 (time 1746626936 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:08:58 (1746626938) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:09:12 (1746626952) targets are mounted 10:09:12 (1746626952) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1746626959 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (34s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 10:09:24 (1746626964) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00152279 s, 2.7 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00331648 s, 1.2 MB/s PASS 41 (2s) == replay-single test 42: recovery after ost failure ===== 10:09:26 (1746626966) total: 800 open/close in 2.54 seconds: 314.82 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1746626973 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second debug=-1 Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 10:09:37 (1746626977) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 10:09:52 (1746626992) targets are mounted 10:09:52 (1746626992) facet_failover done wait for MDS to timeout and recover - unlinked 0 (time 1746627033 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (69s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 10:10:35 (1746627035) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1285408 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:10:38 (1746627038) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:10:52 (1746627052) targets are mounted 10:10:52 (1746627052) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (34s) == replay-single test 44a: race in target handle connect ========================================================== 10:11:09 (1746627069) at_max=40 1 of 10 (1746627071) service : cur 5 worst 5 (at 1746625522, 1549s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 2 of 10 (1746627072) service : cur 5 worst 5 (at 1746625522, 1551s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 3 of 10 (1746627077) service : cur 5 worst 5 (at 1746625522, 1556s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 4 of 10 (1746627083) service : cur 5 worst 5 (at 1746625522, 1562s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 5 of 10 (1746627088) service : cur 5 worst 5 (at 1746625522, 1567s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 6 of 10 (1746627094) service : cur 5 worst 5 (at 1746625522, 1573s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 7 of 10 (1746627099) service : cur 5 worst 5 (at 1746625522, 1578s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 8 of 10 (1746627105) service : cur 5 worst 5 (at 1746625522, 1584s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 9 of 10 (1746627110) service : cur 5 worst 5 (at 1746625522, 1589s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre 10 of 10 (1746627116) service : cur 5 worst 5 (at 1746625522, 1595s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (54s) == replay-single test 44b: race in target handle connect ========================================================== 10:12:03 (1746627123) 1 of 10 (1746627123) service : cur 5 worst 5 (at 1746625522, 1602s ago) 5 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 2 of 10 (1746627164) service : cur 40 worst 40 (at 1746627164, 1s ago) 40 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 3 of 10 (1746627184) service : cur 40 worst 40 (at 1746627164, 21s ago) 1 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 4 of 10 (1746627205) service : cur 40 worst 40 (at 1746627164, 42s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 5 of 10 (1746627225) service : cur 40 worst 40 (at 1746627164, 62s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 6 of 10 (1746627246) service : cur 40 worst 40 (at 1746627164, 83s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 7 of 10 (1746627266) service : cur 40 worst 40 (at 1746627164, 103s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 8 of 10 (1746627287) service : cur 40 worst 40 (at 1746627164, 124s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 9 of 10 (1746627307) service : cur 40 worst 40 (at 1746627164, 144s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre 10 of 10 (1746627328) service : cur 40 worst 40 (at 1746627164, 165s ago) 40 40 40 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 3136 7210904 1% /mnt/lustre PASS 44b (226s) == replay-single test 44c: race in target handle connect ========================================================== 10:15:49 (1746627349) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1828 1285860 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre total: 100 create in 0.26 seconds: 377.58 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:16:08 (1746627368) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:20 (1746627380) targets are mounted 10:16:20 (1746627380) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (39s) == replay-single test 45: Handle failed close ============ 10:16:28 (1746627388) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /mnt/lustre/f45.replay-single has type file OK PASS 45 (2s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 10:16:30 (1746627390) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:16:48 (1746627408) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:17:01 (1746627421) targets are mounted 10:17:01 (1746627421) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (39s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:17:09 (1746627429) total: 20 open/close in 0.07 seconds: 267.05 ops/second Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 10:17:10 (1746627430) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 10:17:24 (1746627444) targets are mounted 10:17:24 (1746627444) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.06 seconds: 339.95 ops/second - unlinked 0 (time 1746627509 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (81s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:18:30 (1746627510) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre total: 20 open/close in 0.07 seconds: 270.44 ops/second Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:18:34 (1746627514) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:18:47 (1746627527) targets are mounted 10:18:47 (1746627527) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.07 seconds: 270.97 ops/second - unlinked 0 (time 1746627591 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (83s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 10:19:53 (1746627593) PASS 50 (7s) == replay-single test 52: time out lock replay (3764) ==== 10:20:00 (1746627600) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7636 fail_loc=0x80000157 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:20:02 (1746627602) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:20:15 (1746627615) targets are mounted 10:20:15 (1746627615) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (79s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 10:21:19 (1746627679) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:21:23 (1746627683) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:21:37 (1746627697) targets are mounted 10:21:37 (1746627697) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (26s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 10:21:45 (1746627705) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:21:49 (1746627709) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:22:03 (1746627723) targets are mounted 10:22:03 (1746627723) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (26s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 10:22:11 (1746627731) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:22:15 (1746627735) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:22:29 (1746627749) targets are mounted 10:22:29 (1746627749) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (26s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 10:22:37 (1746627757) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:22:40 (1746627760) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:22:53 (1746627773) targets are mounted 10:22:53 (1746627773) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (23s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 10:23:00 (1746627780) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:23:05 (1746627785) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:23:18 (1746627798) targets are mounted 10:23:18 (1746627798) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (25s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 10:23:26 (1746627806) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:23:30 (1746627810) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:23:44 (1746627824) targets are mounted 10:23:44 (1746627824) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (25s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:23:51 (1746627831) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:23:56 (1746627836) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:24:10 (1746627850) targets are mounted 10:24:10 (1746627850) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (26s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:24:17 (1746627857) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:24:22 (1746627862) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:24:36 (1746627876) targets are mounted 10:24:36 (1746627876) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (26s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:24:43 (1746627883) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single'rm: : No such file or directory cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (62s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:25:45 (1746627945) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:25:48 (1746627948) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:26:02 (1746627962) targets are mounted 10:26:02 (1746627962) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (34s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:26:19 (1746627979) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:26:23 (1746627983) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:26:37 (1746627997) targets are mounted 10:26:37 (1746627997) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg148-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (28s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:26:47 (1746628007) fail_loc=0x8000012c total: 2500 open/close in 4.68 seconds: 534.67 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1285172 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:26:56 (1746628016) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:27:10 (1746628030) targets are mounted 10:27:10 (1746628030) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1746628042 ; total 0 ; last 0) total: 2500 unlinks in 2 seconds: 1250.000000 unlinks/second PASS 58a (39s) == replay-single test 58b: test replay of setxattr op ==== 10:27:26 (1746628046) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:27:30 (1746628050) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:27:44 (1746628064) targets are mounted 10:27:44 (1746628064) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) oleg148-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (27s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:27:53 (1746628073) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 58c (129s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:30:03 (1746628203) total: 200 open/close in 0.50 seconds: 398.56 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1285336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1746628207 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:30:09 (1746628209) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:30:24 (1746628224) targets are mounted 10:30:24 (1746628224) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1746628230 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (30s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:30:33 (1746628233) total: 800 open/close in 2.99 seconds: 267.87 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2364 1285324 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1285656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre - unlinked 0 (time 1746628240 ; total 0 ; last 0) total: 800 unlinks in 2 seconds: 400.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 10:30:44 (1746628244) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 10:31:00 (1746628260) targets are mounted 10:31:00 (1746628260) facet_failover done Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 10:31:11 (1746628271) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 10:31:25 (1746628285) targets are mounted 10:31:25 (1746628285) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (87s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:32:00 (1746628320) fail_loc=0x8000013a Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:32:02 (1746628322) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:32:15 (1746628335) targets are mounted 10:32:15 (1746628335) facet_failover done Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:32:26 (1746628346) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:32:39 (1746628359) targets are mounted 10:32:39 (1746628359) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00235449 s, 1.7 MB/s PASS 61b (46s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:32:46 (1746628366) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 10:32:58 (1746628378) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 10:33:12 (1746628392) targets are mounted 10:33:12 (1746628392) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (31s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:33:17 (1746628397) Stopping /mnt/lustre-mds1 (opts:) on oleg148-server fail_loc=0x80000605 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg148-client: oleg148-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (8s) == replay-single test 62: don't mis-drop resent replay === 10:33:25 (1746628405) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1285336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2012 1285676 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3140 7210900 1% /mnt/lustre total: 25 open/close in 0.06 seconds: 394.19 ops/second fail_loc=0x80000707 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:33:30 (1746628410) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:33:45 (1746628425) targets are mounted 10:33:45 (1746628425) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1746628488 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (84s) == replay-single test 65a: AT: verify early replies ====== 10:34:49 (1746628489) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:1.0:1746628519.056544:0:7151:0:(client.c:536:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a5aebb80 x1831469575316864/t0(0) o101->lustre-MDT0000-mdc-ffff8800a972f000@192.168.201.148@tcp:12/10 lens 664/66320 e 1 to 0 dl 1746628554 ref 2 fl Rpc:PQr/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 portal 12 : cur 36 worst 40 (at 1746627164, 1360s ago) 36 40 40 40 portal 29 : cur 5 worst 5 (at 1746625792, 2732s ago) 5 0 0 0 portal 23 : cur 5 worst 5 (at 1746625792, 2732s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1746625795, 2729s ago) 5 0 0 5 portal 17 : cur 5 worst 5 (at 1746625871, 2653s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1746626040, 2484s ago) 5 0 0 0 portal 13 : cur 5 worst 5 (at 1746626232, 2292s ago) 5 5 0 0 portal 12 : cur 11 worst 40 (at 1746627164, 1369s ago) 36 40 40 40 portal 29 : cur 5 worst 5 (at 1746625792, 2741s ago) 5 0 0 0 portal 23 : cur 5 worst 5 (at 1746625792, 2741s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1746625795, 2738s ago) 5 0 0 5 portal 17 : cur 5 worst 5 (at 1746625871, 2662s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1746626040, 2493s ago) 5 0 0 0 portal 13 : cur 5 worst 5 (at 1746626232, 2301s ago) 5 5 0 0 PASS 65a (45s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:35:34 (1746628534) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:3.0:1746628564.160402:0:2009:0:(client.c:536:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff88012e1a5c00 x1831469575336576/t0(0) o4->lustre-OST0000-osc-ffff8800a972f000@192.168.201.148@tcp:6/4 lens 4584/448 e 1 to 0 dl 1746628599 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 portal 28 : cur 5 worst 5 (at 1746625792, 2777s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1746625793, 2776s ago) 5 5 5 5 portal 17 : cur 5 worst 5 (at 1746625869, 2700s ago) 5 0 0 5 portal 6 : cur 36 worst 36 (at 1746628569, 0s ago) 36 5 0 0 PASS 65b (36s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:36:10 (1746628570) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1746627164, 1429s ago) 36 40 40 40 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1746627164, 1434s ago) 36 40 40 40 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 36 worst 40 (at 1746627164, 1445s ago) 36 40 40 40 fail_loc=0 portal 12 : cur 5 worst 40 (at 1746627164, 1454s ago) 36 40 40 40 Current MDT timeout 5, worst 40 PASS 66a (49s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:36:59 (1746628619) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (84s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:38:23 (1746628703) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (73s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:39:36 (1746628776) at_history=8 at_history=8 Creating to objid 5057 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 17 open/close in 0.04 seconds: 396.24 ops/second Connected clients: oleg148-client.virtnet oleg148-client.virtnet service : cur 5 worst 5 (at 1746625533, 3267s ago) 1 1 1 1 phase 2 1 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg148-client.virtnet oleg148-client.virtnet service : cur 5 worst 5 (at 1746625533, 3268s ago) 1 1 1 1 0 osc reconnect attempts on 2nd slow PASS 67b (26s) == replay-single test 68: AT: verify slowing locks ======= 10:40:02 (1746628802) at_history=8 at_history=8 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1975: $ldlm_enqueue_min: ambiguous redirect fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1990: $ldlm_enqueue_min: ambiguous redirect PASS 68 (68s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 10:41:10 (1746628870) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (1s) == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 10:41:11 (1746628871) Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg148-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13450 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2168 1285520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1976 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3556156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28120 7149116 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:41:18 (1746628878) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:41:32 (1746628892) targets are mounted 10:41:32 (1746628892) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2168 1285520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1888 3590884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26536 3569716 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28424 7160600 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:41:44 (1746628904) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server oleg148-client.virtnet: looking for dbench program oleg148-client.virtnet: /usr/bin/dbench oleg148-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg148-client.virtnet oleg148-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg148-client.virtnet' oleg148-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg148-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg148-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg148-client.virtnet at Wed May 7 10:41:12 EDT 2025 oleg148-client.virtnet: waiting for dbench pid 13475 oleg148-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg148-client.virtnet: oleg148-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg148-client.virtnet: failed to create barrier semaphore oleg148-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg148-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg148-client.virtnet: releasing clients oleg148-client.virtnet: 1 603 20.97 MB/sec warmup 1 sec latency 11.730 ms oleg148-client.virtnet: 1 1201 11.52 MB/sec warmup 2 sec latency 9.857 ms oleg148-client.virtnet: 1 1849 8.79 MB/sec warmup 3 sec latency 11.437 ms oleg148-client.virtnet: 1 2577 7.24 MB/sec warmup 4 sec latency 8.837 ms oleg148-client.virtnet: 1 3768 7.95 MB/sec warmup 5 sec latency 8.676 ms oleg148-client.virtnet: 1 4309 6.74 MB/sec warmup 6 sec latency 9.988 ms oleg148-client.virtnet: 1 4955 6.45 MB/sec warmup 7 sec latency 17.429 ms oleg148-client.virtnet: 1 5538 5.70 MB/sec warmup 8 sec latency 10.884 ms oleg148-client.virtnet: 1 6325 5.68 MB/sec warmup 9 sec latency 11.507 ms oleg148-client.virtnet: 1 7320 5.89 MB/sec warmup 10 sec latency 10.574 ms oleg148-client.virtnet: 1 7730 5.38 MB/sec warmup 11 sec latency 13.775 ms oleg148-client.virtnet: 1 8329 5.11 MB/sec warmup 12 sec latency 11.403 ms oleg148-client.virtnet: 1 8969 4.97 MB/sec warmup 13 sec latency 10.118 ms oleg148-client.virtnet: 1 9698 4.80 MB/sec warmup 14 sec latency 9.011 ms oleg148-client.virtnet: 1 10786 5.13 MB/sec warmup 15 sec latency 9.213 ms oleg148-client.virtnet: 1 11268 4.89 MB/sec warmup 16 sec latency 16.733 ms oleg148-client.virtnet: 1 11857 4.73 MB/sec warmup 17 sec latency 11.751 ms oleg148-client.virtnet: 1 12481 4.65 MB/sec warmup 18 sec latency 10.985 ms oleg148-client.virtnet: 1 13156 4.53 MB/sec warmup 19 sec latency 9.199 ms oleg148-client.virtnet: 1 14292 4.80 MB/sec warmup 20 sec latency 9.948 ms oleg148-client.virtnet: 1 14845 4.64 MB/sec warmup 21 sec latency 11.085 ms oleg148-client.virtnet: 1 15348 4.52 MB/sec warmup 22 sec latency 12.175 ms oleg148-client.virtnet: 1 15910 4.47 MB/sec warmup 23 sec latency 12.704 ms oleg148-client.virtnet: 1 16573 4.35 MB/sec warmup 24 sec latency 8.220 ms oleg148-client.virtnet: 1 17623 4.52 MB/sec warmup 25 sec latency 11.555 ms oleg148-client.virtnet: 1 18022 4.47 MB/sec warmup 26 sec latency 15.203 ms oleg148-client.virtnet: 1 18410 4.32 MB/sec warmup 27 sec latency 14.234 ms oleg148-client.virtnet: 1 18743 4.20 MB/sec warmup 28 sec latency 16.823 ms oleg148-client.virtnet: 1 19232 4.19 MB/sec warmup 29 sec latency 302.177 ms oleg148-client.virtnet: 1 19552 4.06 MB/sec warmup 30 sec latency 437.956 ms oleg148-client.virtnet: 1 20158 3.99 MB/sec warmup 31 sec latency 13.603 ms oleg148-client.virtnet: 1 20360 3.89 MB/sec warmup 32 sec latency 587.307 ms oleg148-client.virtnet: 1 20360 3.78 MB/sec warmup 33 sec latency 1587.514 ms oleg148-client.virtnet: 1 20360 3Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:41:58 (1746628918) targets are mounted 10:41:58 (1746628918) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13248 3591580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36624 3569984 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49872 7161564 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:42:12 (1746628932) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:42:26 (1746628946) targets are mounted 10:42:26 (1746628946) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec .66 MB/sec warmup 34 sec latency 2587.676 ms oleg148-client.virtnet: 1 20360 3.56 MB/sec warmup 35 sec latency 3587.894 ms oleg148-client.virtnet: 1 20360 3.46 MB/sec warmup 36 sec latency 4588.123 ms oleg148-client.virtnet: 1 20360 3.37 MB/sec warmup 37 sec latency 5588.366 ms oleg148-client.virtnet: 1 20360 3.28 MB/sec warmup 38 sec latency 6588.567 ms oleg148-client.virtnet: 1 20360 3.19 MB/sec warmup 39 sec latency 7588.779 ms oleg148-client.virtnet: 1 20360 3.11 MB/sec warmup 40 sec latency 8589.039 ms oleg148-client.virtnet: 1 20360 3.04 MB/sec warmup 41 sec latency 9589.241 ms oleg148-client.virtnet: 1 20360 2.97 MB/sec warmup 42 sec latency 10589.536 ms oleg148-client.virtnet: 1 20360 2.90 MB/sec warmup 43 sec latency 11589.806 ms oleg148-client.virtnet: 1 20360 2.83 MB/sec warmup 44 sec latency 12589.999 ms oleg148-client.virtnet: 1 20360 2.77 MB/sec warmup 45 sec latency 13590.210 ms oleg148-client.virtnet: 1 20360 2.71 MB/sec warmup 46 sec latency 14590.376 ms oleg148-client.virtnet: 1 20360 2.65 MB/sec warmup 47 sec latency 15590.566 ms oleg148-client.virtnet: 1 20360 2.60 MB/sec warmup 48 sec latency 16590.750 ms oleg148-client.virtnet: 1 20360 2.54 MB/sec warmup 49 sec latency 17590.967 ms oleg148-client.virtnet: 1 20360 2.49 MB/sec warmup 50 sec latency 18591.110 ms oleg148-client.virtnet: 1 21144 2.57 MB/sec warmup 51 sec latency 18706.256 ms oleg148-client.virtnet: 1 21590 2.60 MB/sec warmup 52 sec latency 16.693 ms oleg148-client.virtnet: 1 21831 2.56 MB/sec warmup 53 sec latency 17.626 ms oleg148-client.virtnet: 1 22242 2.53 MB/sec warmup 54 sec latency 18.898 ms oleg148-client.virtnet: 1 22592 2.51 MB/sec warmup 55 sec latency 14.021 ms oleg148-client.virtnet: 1 22992 2.52 MB/sec warmup 56 sec latency 17.348 ms oleg148-client.virtnet: 1 23310 2.48 MB/sec warmup 57 sec latency 17.301 ms oleg148-client.virtnet: 1 23766 2.46 MB/sec warmup 58 sec latency 17.051 ms oleg148-client.virtnet: 1 24347 2.51 MB/sec warmup 59 sec latency 19.723 ms oleg148-client.virtnet: 1 25325 1.28 MB/sec execute 1 sec latency 17.662 ms oleg148-client.virtnet: 1 25633 0.83 MB/sec execute 2 sec latency 21.157 ms oleg148-client.virtnet: 1 26018 1.11 MB/sec execute 3 sec latency 17.576 ms oleg148-client.virtnet: 1 26435 1.61 MB/sec execute 4 sec latency 14.841 ms oleg148-client.virtnet: 1 26855 1.36 MB/sec execute 5 sec latency 18.824 ms oleg148-client.virtnet: 1 27450 1.55 MB/sec execute 6 sec latency 11.243 ms oleg148-client.virtnet: 1 28514 2.71 MB/sec execute 7 sec latency 10.014 ms oleg148-client.virtnet: 1 28881 2.55 MB/sec execute 8 sec latency 14.478 ms oleg148-client.virtnet: 1 29409 2.37 MB/sec execute 9 sec latency 11.363 ms oleg148-client.virtnet: 1 30076 2.57 MB/sec execute 10 sec latency 9.558 ms oleg148-client.virtnet: 1 30720 2.47 MB/sec execute 11 sec latency 11.409 ms oleg148-client.virtnet: 1 31604 2.81 MB/sec execute 12 sec latency 10.642 ms oleg148-client.virtnet: 1 32258 3.02 MB/sec execute 13 sec latency 12.896 ms oleg148-client.virtnet: 1 32764 2.86 MB/sec execute 14 sec latency 12.378 ms oleg148-client.virtnet: 1 33362 2.97 MB/sec execute 15 sec latency 12.128 ms oleg148-client.virtnet: 1 33846 2.81 MB/sec execute 16 sec latency 11.541 ms oleg148-client.virtnet: 1 34439 2.78 MB/sec execute 17 sec latency 11.827 ms oleg148-client.virtnet: 1 35529 3.17 MB/sec execute 18 sec latency 11.328 ms oleg148-client.virtnet: 1 36023 3.09 MB/sec execute 19 sec latency 9.241 ms oleg148-client.virtnet: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13336 3592656 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3570092 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49724 7162748 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:42:37 (1746628957) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:42:52 (1746628972) targets are mounted 10:42:52 (1746628972) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3590476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36708 3570128 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160604 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:43:05 (1746628985) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 1 36572 3.03 MB/sec execute 20 sec latency 12.787 ms oleg148-client.virtnet: 1 37203 3.04 MB/sec execute 21 sec latency 12.328 ms oleg148-client.virtnet: 1 37854 2.98 MB/sec execute 22 sec latency 86.193 ms oleg148-client.virtnet: 1 38422 3.09 MB/sec execute 23 sec latency 463.814 ms oleg148-client.virtnet: 1 39323 3.23 MB/sec execute 24 sec latency 10.713 ms oleg148-client.virtnet: 1 39558 3.11 MB/sec execute 25 sec latency 493.515 ms oleg148-client.virtnet: 1 39558 2.99 MB/sec execute 26 sec latency 1493.695 ms oleg148-client.virtnet: 1 39558 2.88 MB/sec execute 27 sec latency 2493.881 ms oleg148-client.virtnet: 1 39558 2.78 MB/sec execute 28 sec latency 3494.055 ms oleg148-client.virtnet: 1 39558 2.68 MB/sec execute 29 sec latency 4494.289 ms oleg148-client.virtnet: 1 39558 2.59 MB/sec execute 30 sec latency 5494.554 ms oleg148-client.virtnet: 1 39558 2.51 MB/sec execute 31 sec latency 6494.810 ms oleg148-client.virtnet: 1 39558 2.43 MB/sec execute 32 sec latency 7495.134 ms oleg148-client.virtnet: 1 39558 2.36 MB/sec execute 33 sec latency 8495.381 ms oleg148-client.virtnet: 1 39558 2.29 MB/sec execute 34 sec latency 9495.739 ms oleg148-client.virtnet: 1 39558 2.22 MB/sec execute 35 sec latency 10496.137 ms oleg148-client.virtnet: 1 39558 2.16 MB/sec execute 36 sec latency 11496.398 ms oleg148-client.virtnet: 1 39558 2.10 MB/sec execute 37 sec latency 12496.577 ms oleg148-client.virtnet: 1 39558 2.05 MB/sec execute 38 sec latency 13496.776 ms oleg148-client.virtnet: 1 39558 1.99 MB/sec execute 39 sec latency 14497.038 ms oleg148-client.virtnet: 1 39558 1.94 MB/sec execute 40 sec latency 15497.309 ms oleg148-client.virtnet: 1 39558 1.90 MB/sec execute 41 sec latency 16497.595 ms oleg148-client.virtnet: 1 39558 1.85 MB/sec execute 42 sec latency 17497.934 ms oleg148-client.virtnet: 1 39558 1.81 MB/sec execute 43 sec latency 18498.179 ms oleg148-client.virtnet: 1 39632 1.77 MB/sec execute 44 sec latency 19293.279 ms oleg148-client.virtnet: 1 40036 1.75 MB/sec execute 45 sec latency 16.556 ms oleg148-client.virtnet: 1 40510 1.80 MB/sec execute 46 sec latency 16.977 ms oleg148-client.virtnet: 1 40857 1.77 MB/sec execute 47 sec latency 15.360 ms oleg148-client.virtnet: 1 41409 1.76 MB/sec execute 48 sec latency 11.276 ms oleg148-client.virtnet: 1 42037 1.84 MB/sec execute 49 sec latency 14.853 ms oleg148-client.virtnet: 1 42720 1.91 MB/sec execute 50 sec latency 14.054 ms oleg148-client.virtnet: 1 42986 1.90 MB/sec execute 51 sec latency 17.444 ms oleg148-client.virtnet: 1 43295 1.87 MB/sec execute 52 sec latency 20.732 ms oleg148-client.virtnet: 1 43657 1.86 MB/sec execute 53 sec latency 18.721 ms oleg148-client.virtnet: 1 44059 1.89 MB/sec execute 54 sec latency 18.707 ms oleg148-client.virtnet: 1 44332 1.86 MB/sec execute 55 sec latency 20.299 ms oleg148-client.virtnet: 1 44660 1.83 MB/sec execute 56 sec latency 13.186 ms oleg148-client.virtnet: 1 45052 1.83 MB/sec execute 57 sec latency 16.818 ms oleg148-client.virtnet: 1 45784 1.90 MB/sec execute 58 sec latency 18.424 ms oleg148-client.virtnet: 1 46433 1.96 MB/sec execute 59 sec latency 14.539 ms oleg148-client.virtnet: 1 46772 1.93 MB/sec execute 60 sec latency 16.129 ms oleg148-client.virtnet: 1 47290 1.93 MB/sec execute 61 sec latency 12.738 ms oleg148-client.virtnet: 1 47815 1.96 MB/sec execute 62 sec latency 14.780 ms oleg148-client.virtnet: 1 48333 1.93 MB/sec execute 63 sec latency 18.420 ms oleg148-client.virtnet: 1 49121 2.01 MB/sec execute 64 sStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:43:19 (1746628999) targets are mounted 10:43:19 (1746628999) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13064 3592964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36560 3569728 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162692 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:43:32 (1746629012) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:43:46 (1746629026) targets are mounted 10:43:46 (1746629026) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13608 3592640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3567880 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49996 7160520 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:43:58 (1746629038) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server ec latency 12.565 ms oleg148-client.virtnet: 1 49847 2.06 MB/sec execute 65 sec latency 13.095 ms oleg148-client.virtnet: 1 50157 2.05 MB/sec execute 66 sec latency 14.882 ms oleg148-client.virtnet: 1 50569 2.03 MB/sec execute 67 sec latency 14.847 ms oleg148-client.virtnet: 1 51075 2.06 MB/sec execute 68 sec latency 18.323 ms oleg148-client.virtnet: 1 51471 2.04 MB/sec execute 69 sec latency 13.502 ms oleg148-client.virtnet: 1 52068 2.03 MB/sec execute 70 sec latency 8.718 ms oleg148-client.virtnet: 1 52683 2.08 MB/sec execute 71 sec latency 18.669 ms oleg148-client.virtnet: 1 53299 2.12 MB/sec execute 72 sec latency 17.480 ms oleg148-client.virtnet: 1 53622 2.11 MB/sec execute 73 sec latency 16.367 ms oleg148-client.virtnet: 1 53978 2.09 MB/sec execute 74 sec latency 15.734 ms oleg148-client.virtnet: 1 54479 2.08 MB/sec execute 75 sec latency 15.447 ms oleg148-client.virtnet: 1 55063 2.10 MB/sec execute 76 sec latency 11.456 ms oleg148-client.virtnet: 1 55461 2.08 MB/sec execute 77 sec latency 13.527 ms oleg148-client.virtnet: 1 55870 2.09 MB/sec execute 78 sec latency 17.559 ms oleg148-client.virtnet: 1 56613 2.14 MB/sec execute 79 sec latency 19.217 ms oleg148-client.virtnet: 1 56855 2.15 MB/sec execute 80 sec latency 678.944 ms oleg148-client.virtnet: 1 56855 2.12 MB/sec execute 81 sec latency 1679.285 ms oleg148-client.virtnet: 1 56855 2.10 MB/sec execute 82 sec latency 2679.525 ms oleg148-client.virtnet: 1 56855 2.07 MB/sec execute 83 sec latency 3679.803 ms oleg148-client.virtnet: 1 56855 2.05 MB/sec execute 84 sec latency 4680.016 ms oleg148-client.virtnet: 1 56855 2.02 MB/sec execute 85 sec latency 5680.251 ms oleg148-client.virtnet: 1 56855 2.00 MB/sec execute 86 sec latency 6680.442 ms oleg148-client.virtnet: 1 56855 1.98 MB/sec execute 87 sec latency 7680.665 ms oleg148-client.virtnet: 1 56855 1.95 MB/sec execute 88 sec latency 8680.824 ms oleg148-client.virtnet: 1 56855 1.93 MB/sec execute 89 sec latency 9680.977 ms oleg148-client.virtnet: 1 56855 1.91 MB/sec execute 90 sec latency 10681.174 ms oleg148-client.virtnet: 1 56855 1.89 MB/sec execute 91 sec latency 11681.327 ms oleg148-client.virtnet: 1 56855 1.87 MB/sec execute 92 sec latency 12681.515 ms oleg148-client.virtnet: 1 56855 1.85 MB/sec execute 93 sec latency 13681.662 ms oleg148-client.virtnet: 1 56855 1.83 MB/sec execute 94 sec latency 14681.811 ms oleg148-client.virtnet: 1 56855 1.81 MB/sec execute 95 sec latency 15681.965 ms oleg148-client.virtnet: 1 56855 1.79 MB/sec execute 96 sec latency 16682.123 ms oleg148-client.virtnet: 1 56855 1.77 MB/sec execute 97 sec latency 17682.321 ms oleg148-client.virtnet: 1 56877 1.75 MB/sec execute 98 sec latency 18622.559 ms oleg148-client.virtnet: 1 57351 1.75 MB/sec execute 99 sec latency 8.490 ms oleg148-client.virtnet: 1 57902 1.75 MB/sec execute 100 sec latency 12.252 ms oleg148-client.virtnet: 1 58465 1.77 MB/sec execute 101 sec latency 13.682 ms oleg148-client.virtnet: 1 59158 1.77 MB/sec execute 102 sec latency 9.654 ms oleg148-client.virtnet: 1 60154 1.82 MB/sec execute 103 sec latency 11.227 ms oleg148-client.virtnet: 1 60778 1.85 MB/sec execute 104 sec latency 11.135 ms oleg148-client.virtnet: 1 61323 1.84 MB/sec execute 105 sec latency 10.949 ms oleg148-client.virtnet: 1 61980 1.87 MB/sec execute 106 sec latency 14.719 ms oleg148-client.virtnet: 1 62487 1.85 MB/sec execute 107 sec latency 20.092 ms oleg148-client.virtnet: 1 63260 1.90 MB/sec execute 108 sec latency 43.815 ms oleg148-client.virtnet: Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:44:12 (1746629052) targets are mounted 10:44:12 (1746629052) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13428 3592448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3570104 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49816 7162552 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:44:24 (1746629064) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:44:38 (1746629078) targets are mounted 10:44:38 (1746629078) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 64074 1.94 MB/sec execute 109 sec latency 12.536 ms oleg148-client.virtnet: 1 64576 1.93 MB/sec execute 110 sec latency 11.253 ms oleg148-client.virtnet: 1 65097 1.93 MB/sec execute 111 sec latency 12.492 ms oleg148-client.virtnet: 1 65678 1.94 MB/sec execute 112 sec latency 12.856 ms oleg148-client.virtnet: 1 66308 1.94 MB/sec execute 113 sec latency 9.728 ms oleg148-client.virtnet: 1 67377 2.01 MB/sec execute 114 sec latency 13.332 ms oleg148-client.virtnet: 1 67892 2.01 MB/sec execute 115 sec latency 10.925 ms oleg148-client.virtnet: 1 68487 2.01 MB/sec execute 116 sec latency 12.182 ms oleg148-client.virtnet: 1 69113 2.02 MB/sec execute 117 sec latency 10.932 ms oleg148-client.virtnet: 1 69802 2.01 MB/sec execute 118 sec latency 9.835 ms oleg148-client.virtnet: 1 70707 2.06 MB/sec execute 119 sec latency 16.323 ms oleg148-client.virtnet: 1 71357 2.08 MB/sec execute 120 sec latency 8.834 ms oleg148-client.virtnet: 1 71891 2.07 MB/sec execute 121 sec latency 10.420 ms oleg148-client.virtnet: 1 72474 2.09 MB/sec execute 122 sec latency 11.569 ms oleg148-client.virtnet: 1 73059 2.08 MB/sec execute 123 sec latency 10.591 ms oleg148-client.virtnet: 1 73873 2.11 MB/sec execute 124 sec latency 11.846 ms oleg148-client.virtnet: 1 74682 2.14 MB/sec execute 125 sec latency 10.891 ms oleg148-client.virtnet: 1 75023 2.13 MB/sec execute 126 sec latency 14.831 ms oleg148-client.virtnet: 1 75486 2.12 MB/sec execute 127 sec latency 13.919 ms oleg148-client.virtnet: 1 76043 2.14 MB/sec execute 128 sec latency 13.468 ms oleg148-client.virtnet: 1 76287 2.13 MB/sec execute 129 sec latency 450.073 ms oleg148-client.virtnet: 1 76947 2.12 MB/sec execute 130 sec latency 9.155 ms oleg148-client.virtnet: 1 77882 2.16 MB/sec execute 131 sec latency 13.828 ms oleg148-client.virtnet: 1 77882 2.15 MB/sec execute 132 sec latency 1008.468 ms oleg148-client.virtnet: 1 77882 2.13 MB/sec execute 133 sec latency 2008.632 ms oleg148-client.virtnet: 1 77882 2.12 MB/sec execute 134 sec latency 3008.795 ms oleg148-client.virtnet: 1 77882 2.10 MB/sec execute 135 sec latency 4008.995 ms oleg148-client.virtnet: 1 77882 2.08 MB/sec execute 136 sec latency 5009.152 ms oleg148-client.virtnet: 1 77882 2.07 MB/sec execute 137 sec latency 6009.301 ms oleg148-client.virtnet: 1 77882 2.05 MB/sec execute 138 sec latency 7009.431 ms oleg148-client.virtnet: 1 77882 2.04 MB/sec execute 139 sec latency 8009.575 ms oleg148-client.virtnet: 1 77882 2.02 MB/sec execute 140 sec latency 9009.745 ms oleg148-client.virtnet: 1 77882 2.01 MB/sec execute 141 sec latency 10009.922 ms oleg148-client.virtnet: 1 77882 2.00 MB/sec execute 142 sec latency 11010.055 ms oleg148-client.virtnet: 1 77882 1.98 MB/sec execute 143 sec latency 12010.219 ms oleg148-client.virtnet: 1 77882 1.97 MB/sec execute 144 sec latency 13010.428 ms oleg148-client.virtnet: 1 77882 1.95 MB/sec execute 145 sec latency 14010.610 ms oleg148-client.virtnet: 1 77882 1.94 MB/sec execute 146 sec latency 15010.770 ms oleg148-client.virtnet: 1 77882 1.93 MB/sec execute 147 sec latency 16010.921 ms oleg148-client.virtnet: 1 77882 1.92 MB/sec execute 148 sec latency 17011.120 ms oleg148-client.virtnet: 1 77882 1.90 MB/sec execute 149 sec latency 18011.330 ms oleg148-client.virtnet: 1 78318 1.92 MB/sec execute 150 sec latency 18412.577 ms oleg148-client.virtnet: 1 78795 1.91 MB/sec execute 151 sec latency 14.472 ms oleg148-client.virtnet: 1 79328 1.91 MB/sec execute 152 sec latency 11.986 ms oleg148-client.virtnet: 1 79921 1.92 MB/sec execute 153 sec lUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3593748 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3569900 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163648 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:44:49 (1746629089) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:45:03 (1746629103) targets are mounted 10:45:03 (1746629103) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13176 3592828 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36524 3569908 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49700 7162736 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:45:15 (1746629115) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:45:29 (1746629129) targets are mounted 10:45:29 (1746629129) facet_failover done atency 12.769 ms oleg148-client.virtnet: 1 80543 1.92 MB/sec execute 154 sec latency 11.288 ms oleg148-client.virtnet: 1 81647 1.97 MB/sec execute 155 sec latency 9.568 ms oleg148-client.virtnet: 1 82129 1.97 MB/sec execute 156 sec latency 10.507 ms oleg148-client.virtnet: 1 82730 1.97 MB/sec execute 157 sec latency 13.147 ms oleg148-client.virtnet: 1 83410 1.98 MB/sec execute 158 sec latency 9.899 ms oleg148-client.virtnet: 1 84089 1.98 MB/sec execute 159 sec latency 9.027 ms oleg148-client.virtnet: 1 85216 2.03 MB/sec execute 160 sec latency 9.222 ms oleg148-client.virtnet: 1 85770 2.03 MB/sec execute 161 sec latency 8.292 ms oleg148-client.virtnet: 1 86356 2.03 MB/sec execute 162 sec latency 10.824 ms oleg148-client.virtnet: 1 86955 2.03 MB/sec execute 163 sec latency 13.347 ms oleg148-client.virtnet: 1 87579 2.03 MB/sec execute 164 sec latency 9.589 ms oleg148-client.virtnet: 1 88681 2.08 MB/sec execute 165 sec latency 11.022 ms oleg148-client.virtnet: 1 89221 2.08 MB/sec execute 166 sec latency 8.441 ms oleg148-client.virtnet: 1 89834 2.08 MB/sec execute 167 sec latency 12.482 ms oleg148-client.virtnet: 1 90453 2.09 MB/sec execute 168 sec latency 9.928 ms oleg148-client.virtnet: 1 91117 2.09 MB/sec execute 169 sec latency 9.448 ms oleg148-client.virtnet: 1 92120 2.12 MB/sec execute 170 sec latency 10.925 ms oleg148-client.virtnet: 1 92729 2.13 MB/sec execute 171 sec latency 9.394 ms oleg148-client.virtnet: 1 93337 2.13 MB/sec execute 172 sec latency 10.256 ms oleg148-client.virtnet: 1 93964 2.14 MB/sec execute 173 sec latency 12.051 ms oleg148-client.virtnet: 1 94649 2.14 MB/sec execute 174 sec latency 9.457 ms oleg148-client.virtnet: 1 95645 2.17 MB/sec execute 175 sec latency 12.207 ms oleg148-client.virtnet: 1 96263 2.18 MB/sec execute 176 sec latency 9.157 ms oleg148-client.virtnet: 1 96811 2.17 MB/sec execute 177 sec latency 11.584 ms oleg148-client.virtnet: 1 97475 2.19 MB/sec execute 178 sec latency 14.667 ms oleg148-client.virtnet: 1 97902 2.18 MB/sec execute 179 sec latency 16.513 ms oleg148-client.virtnet: 1 98257 2.17 MB/sec execute 180 sec latency 430.816 ms oleg148-client.virtnet: 1 99374 2.22 MB/sec execute 181 sec latency 10.337 ms oleg148-client.virtnet: 1 99857 2.21 MB/sec execute 182 sec latency 10.241 ms oleg148-client.virtnet: 1 99883 2.20 MB/sec execute 183 sec latency 948.043 ms oleg148-client.virtnet: 1 99883 2.19 MB/sec execute 184 sec latency 1948.279 ms oleg148-client.virtnet: 1 99883 2.18 MB/sec execute 185 sec latency 2948.506 ms oleg148-client.virtnet: 1 99883 2.17 MB/sec execute 186 sec latency 3948.831 ms oleg148-client.virtnet: 1 99883 2.15 MB/sec execute 187 sec latency 4949.082 ms oleg148-client.virtnet: 1 99883 2.14 MB/sec execute 188 sec latency 5949.316 ms oleg148-client.virtnet: 1 99883 2.13 MB/sec execute 189 sec latency 6949.541 ms oleg148-client.virtnet: 1 99883 2.12 MB/sec execute 190 sec latency 7949.734 ms oleg148-client.virtnet: 1 99883 2.11 MB/sec execute 191 sec latency 8949.944 ms oleg148-client.virtnet: 1 99883 2.10 MB/sec execute 192 sec latency 9950.181 ms oleg148-client.virtnet: 1 99883 2.09 MB/sec execute 193 sec latency 10950.311 ms oleg148-client.virtnet: 1 99883 2.08 MB/sec execute 194 sec latency 11950.523 ms oleg148-client.virtnet: 1 99883 2.07 MB/sec execute 195 sec latency 12950.692 ms oleg148-client.virtnet: 1 99883 2.06 MB/sec execute 196 sec latency 13950.872 ms oleg148-client.virtnet: 1 99883 2.04 MB/sec execute 197 sec latency 14951.072 ms oleg148-client.virtnet: 1 99883 2.03 MB/soleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3590420 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36708 3569648 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160068 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:45:42 (1746629142) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:45:56 (1746629156) targets are mounted 10:45:56 (1746629156) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12996 3592800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36616 3567272 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160072 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:46:14 (1746629174) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server ec execute 198 sec latency 15951.279 ms oleg148-client.virtnet: 1 99883 2.02 MB/sec execute 199 sec latency 16951.515 ms oleg148-client.virtnet: 1 99883 2.01 MB/sec execute 200 sec latency 17951.718 ms oleg148-client.virtnet: 1 99961 2.00 MB/sec execute 201 sec latency 18636.646 ms oleg148-client.virtnet: 1 100348 2.00 MB/sec execute 202 sec latency 19.230 ms oleg148-client.virtnet: 1 100939 2.01 MB/sec execute 203 sec latency 10.580 ms oleg148-client.virtnet: 1 101349 2.00 MB/sec execute 204 sec latency 19.438 ms oleg148-client.virtnet: 1 101756 2.00 MB/sec execute 205 sec latency 16.096 ms oleg148-client.virtnet: 1 102128 2.01 MB/sec execute 206 sec latency 28.054 ms oleg148-client.virtnet: 1 102569 2.01 MB/sec execute 207 sec latency 25.866 ms oleg148-client.virtnet: 1 102995 2.02 MB/sec execute 208 sec latency 19.575 ms oleg148-client.virtnet: 1 103245 2.02 MB/sec execute 209 sec latency 20.099 ms oleg148-client.virtnet: 1 103421 2.01 MB/sec execute 210 sec latency 39.465 ms oleg148-client.virtnet: 1 103727 2.00 MB/sec execute 211 sec latency 21.145 ms oleg148-client.virtnet: 1 104050 2.00 MB/sec execute 212 sec latency 24.499 ms oleg148-client.virtnet: 1 104449 2.01 MB/sec execute 213 sec latency 17.981 ms oleg148-client.virtnet: 1 104957 2.00 MB/sec execute 214 sec latency 11.577 ms oleg148-client.virtnet: 1 106076 2.02 MB/sec execute 215 sec latency 8.928 ms oleg148-client.virtnet: 1 106735 2.04 MB/sec execute 216 sec latency 10.433 ms oleg148-client.virtnet: 1 107258 2.04 MB/sec execute 217 sec latency 9.460 ms oleg148-client.virtnet: 1 107883 2.05 MB/sec execute 218 sec latency 10.421 ms oleg148-client.virtnet: 1 108400 2.04 MB/sec execute 219 sec latency 12.333 ms oleg148-client.virtnet: 1 109032 2.04 MB/sec execute 220 sec latency 10.022 ms oleg148-client.virtnet: 1 110086 2.08 MB/sec execute 221 sec latency 10.256 ms oleg148-client.virtnet: 1 110532 2.07 MB/sec execute 222 sec latency 10.424 ms oleg148-client.virtnet: 1 111095 2.07 MB/sec execute 223 sec latency 10.676 ms oleg148-client.virtnet: 1 111695 2.08 MB/sec execute 224 sec latency 12.403 ms oleg148-client.virtnet: 1 112266 2.07 MB/sec execute 225 sec latency 9.235 ms oleg148-client.virtnet: 1 113242 2.10 MB/sec execute 226 sec latency 10.110 ms oleg148-client.virtnet: 1 113943 2.11 MB/sec execute 227 sec latency 9.925 ms oleg148-client.virtnet: 1 114488 2.11 MB/sec execute 228 sec latency 11.267 ms oleg148-client.virtnet: 1 115106 2.12 MB/sec execute 229 sec latency 13.762 ms oleg148-client.virtnet: 1 115685 2.11 MB/sec execute 230 sec latency 12.811 ms oleg148-client.virtnet: 1 116527 2.13 MB/sec execute 231 sec latency 12.043 ms oleg148-client.virtnet: 1 117399 2.15 MB/sec execute 232 sec latency 9.884 ms oleg148-client.virtnet: 1 117890 2.14 MB/sec execute 233 sec latency 74.180 ms oleg148-client.virtnet: 1 118277 2.14 MB/sec execute 234 sec latency 361.360 ms oleg148-client.virtnet: 1 118865 2.14 MB/sec execute 235 sec latency 12.843 ms oleg148-client.virtnet: 1 119135 2.14 MB/sec execute 236 sec latency 501.150 ms oleg148-client.virtnet: 1 119135 2.13 MB/sec execute 237 sec latency 1501.338 ms oleg148-client.virtnet: 1 119135 2.12 MB/sec execute 238 sec latency 2501.748 ms oleg148-client.virtnet: 1 119135 2.11 MB/sec execute 239 sec latency 3501.943 ms oleg148-client.virtnet: 1 119135 2.10 MB/sec execute 240 sec latency 4502.122 ms oleg148-client.virtnet: 1 119135 2.09 MB/sec execute 241 sec latency 5502.306 ms oleg148-client.virtnet: 1 119135 2.08 MB/sec execute 242 sec latency 6502.507 ms oleg148-client.virtnet: 1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:46:28 (1746629188) targets are mounted 10:46:28 (1746629188) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 119135 2.07 MB/sec execute 243 sec latency 7502.702 ms oleg148-client.virtnet: 1 119135 2.07 MB/sec execute 244 sec latency 8502.908 ms oleg148-client.virtnet: 1 119135 2.06 MB/sec execute 245 sec latency 9503.102 ms oleg148-client.virtnet: 1 119135 2.05 MB/sec execute 246 sec latency 10503.258 ms oleg148-client.virtnet: 1 119135 2.04 MB/sec execute 247 sec latency 11503.455 ms oleg148-client.virtnet: 1 119135 2.03 MB/sec execute 248 sec latency 12503.673 ms oleg148-client.virtnet: 1 119135 2.02 MB/sec execute 249 sec latency 13503.859 ms oleg148-client.virtnet: 1 119135 2.02 MB/sec execute 250 sec latency 14504.038 ms oleg148-client.virtnet: 1 119135 2.01 MB/sec execute 251 sec latency 15504.192 ms oleg148-client.virtnet: 1 119135 2.00 MB/sec execute 252 sec latency 16504.326 ms oleg148-client.virtnet: 1 119135 1.99 MB/sec execute 253 sec latency 17504.443 ms oleg148-client.virtnet: 1 119135 1.98 MB/sec execute 254 sec latency 18504.639 ms oleg148-client.virtnet: 1 119135 1.98 MB/sec execute 255 sec latency 19504.780 ms oleg148-client.virtnet: 1 119135 1.97 MB/sec execute 256 sec latency 20504.957 ms oleg148-client.virtnet: 1 119135 1.96 MB/sec execute 257 sec latency 21505.136 ms oleg148-client.virtnet: 1 119135 1.95 MB/sec execute 258 sec latency 22505.345 ms oleg148-client.virtnet: 1 119135 1.95 MB/sec execute 259 sec latency 23505.546 ms oleg148-client.virtnet: 1 119253 1.94 MB/sec execute 260 sec latency 24314.340 ms oleg148-client.virtnet: 1 120006 1.96 MB/sec execute 261 sec latency 11.513 ms oleg148-client.virtnet: 1 120862 1.97 MB/sec execute 262 sec latency 11.649 ms oleg148-client.virtnet: 1 121339 1.97 MB/sec execute 263 sec latency 11.734 ms oleg148-client.virtnet: 1 121870 1.97 MB/sec execute 264 sec latency 9.702 ms oleg148-client.virtnet: 1 122496 1.97 MB/sec execute 265 sec latency 11.982 ms oleg148-client.virtnet: 1 123194 1.98 MB/sec execute 266 sec latency 9.286 ms oleg148-client.virtnet: 1 124263 2.00 MB/sec execute 267 sec latency 9.288 ms oleg148-client.virtnet: 1 124764 2.00 MB/sec execute 268 sec latency 9.623 ms oleg148-client.virtnet: 1 125352 2.00 MB/sec execute 269 sec latency 11.143 ms oleg148-client.virtnet: 1 125971 2.01 MB/sec execute 270 sec latency 11.162 ms oleg148-client.virtnet: 1 126629 2.01 MB/sec execute 271 sec latency 10.602 ms oleg148-client.virtnet: 1 127755 2.04 MB/sec execute 272 sec latency 9.446 ms oleg148-client.virtnet: 1 128237 2.04 MB/sec execute 273 sec latency 10.569 ms oleg148-client.virtnet: 1 128822 2.04 MB/sec execute 274 sec latency 12.769 ms oleg148-client.virtnet: 1 129454 2.04 MB/sec execute 275 sec latency 10.694 ms oleg148-client.virtnet: 1 130112 2.04 MB/sec execute 276 sec latency 7.870 ms oleg148-client.virtnet: 1 131105 2.06 MB/sec execute 277 sec latency 10.515 ms oleg148-client.virtnet: 1 131719 2.07 MB/sec execute 278 sec latency 10.065 ms oleg148-client.virtnet: 1 132267 2.06 MB/sec execute 279 sec latency 11.655 ms oleg148-client.virtnet: 1 132868 2.07 MB/sec execute 280 sec latency 11.351 ms oleg148-client.virtnet: 1 133402 2.07 MB/sec execute 281 sec latency 11.555 ms oleg148-client.virtnet: 1 134047 2.08 MB/sec execute 282 sec latency 11.619 ms oleg148-client.virtnet: 1 134917 2.09 MB/sec execute 283 sec latency 10.890 ms oleg148-client.virtnet: 1 135368 2.09 MB/sec execute 284 sec latency 11.202 ms oleg148-client.virtnet: 1 135906 2.09 MB/sec execute 285 sec latency 11.563 ms oleg148-client.virtnet: 1 136479 2.10 MB/sec execute 286 sec latency 10.507 ms oleg148-client.virtnet: 1 137028 2.09 MB/sec execute 287 sec latency 11.060 ms oleg148-client.virtnet: 1 137832 2.11 MB/sec execute 288 sec latency 11.743 ms oleg148-client.virtnet: 1 138625 2.12 MB/sec execute 289 sec latency 10.394 ms oleg148-client.virtnet: 1 139044 2.12 MB/sec execute 290 sec latency 14.514 ms oleg148-client.virtnet: 1 139509 2.11 MB/sec execute 291 sec latency 14.040 ms oleg148-client.virtnet: 1 140079 2.12 MB/sec execute 292 sec latency 13.710 ms oleg148-client.virtnet: 1 140714 2.12 MB/sec execute 293 sec latency 8.019 ms oleg148-client.virtnet: 1 141637 2.13 MB/sec execute 294 sec latency 13.162 ms oleg148-client.virtnet: 1 142298 2.14 MB/sec execute 295 sec latency 9.285 ms oleg148-client.virtnet: 1 142825 2.14 MB/sec execute 296 sec latency 11.933 ms oleg148-client.virtnet: 1 143475 2.15 MB/sec execute 297 sec latency 10.630 ms oleg148-client.virtnet: 1 144076 2.14 MB/sec execute 298 sec latency 10.520 ms oleg148-client.virtnet: 1 144920 2.16 MB/sec execute 299 sec latency 9.883 ms oleg148-client.virtnet: 1 cleanup 300 sec oleg148-client.virtnet: 0 cleanup 300 sec oleg148-client.virtnet: oleg148-client.virtnet: Operation Count AvgLat MaxLat oleg148-client.virtnet: ---------------------------------------- oleg148-client.virtnet: NTCreateX 20917 8.247 24314.318 oleg148-client.virtnet: Close 15338 2.219 18636.618 oleg148-client.virtnet: Rename 884 8.268 27.981 oleg148-client.virtnet: Unlink 4235 1.908 7.997 oleg148-client.virtnet: Qpathinfo 18950 2.453 18412.560 oleg148-client.virtnet: Qfileinfo 3299 0.152 2.330 oleg148-client.virtnet: Qfsinfo 3476 0.233 13.132 oleg148-client.virtnet: Sfileinfo 1700 4.661 16.108 oleg148-client.virtnet: Find 7315 0.354 11.709 oleg148-client.virtnet: WriteX 10327 0.858 11.575 oleg148-client.virtnet: ReadX 32737 0.021 4.483 oleg148-client.virtnet: LockX 68 0.995 2.867 oleg148-client.virtnet: UnlockX 68 1.023 2.048 oleg148-client.virtnet: Flush 1462 6.319 463.794 oleg148-client.virtnet: oleg148-client.virtnet: Throughput 2.15671 MB/sec 1 clients 1 procs max_latency=24314.340 ms oleg148-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg148-client.virtnet at Wed May 7 10:47:12 EDT 2025 with return code 0 oleg148-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg148-client.virtnet oleg148-client.virtnet: /mnt/lustre/d70b.replay-single/oleg148-client.virtnet /mnt/lustre/d70b.replay-single/oleg148-client.virtnet oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg148-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg148-client.virtnet: removed directory: 'clients/client0' oleg148-client.virtnet: removed directory: 'clients' oleg148-client.virtnet: removed 'client.txt' oleg148-client.virtnet: /mnt/lustre/d70b.replay-single/oleg148-client.virtnet oleg148-client.virtnet: dbench successfully finished PASS 70b (363s) == replay-single test 70c: tar 2mdts recovery ============ 10:47:14 (1746629234) Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar 27583 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6976 1280712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4628 1283060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7720 3580652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 10936 3574900 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 18656 7155552 1% /mnt/lustre test_70c fail mds2 1 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 10:49:27 (1746629367) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 10:49:42 (1746629382) targets are mounted 10:49:42 (1746629382) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7372 1280316 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4560 1283128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14132 3568228 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8000 3581048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 22132 7149276 1% /mnt/lustre test_70c fail mds1 2 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:52:01 (1746629521) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:52:16 (1746629536) targets are mounted 10:52:16 (1746629536) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (359s) == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 10:53:13 (1746629593) Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 30870 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4804 1282884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6348 1281340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3604880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3604524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7209404 1% /mnt/lustre test_70d fail mds1 1 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:55:33 (1746629733) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:55:48 (1746629748) targets are mounted 10:55:48 (1746629748) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4944 1282744 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6292 1281396 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3604880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3604524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7209404 1% /mnt/lustre test_70d fail mds1 2 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 10:58:16 (1746629896) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 10:58:30 (1746629910) targets are mounted 10:58:30 (1746629910) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 30870 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (325s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:58:38 (1746629918) debug=+ha Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started PID=17684 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3660 1284028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3604880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3604524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7209404 1% /mnt/lustre test_70e fail mds1 1 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:00:53 (1746630053) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:01:07 (1746630067) targets are mounted 11:01:07 (1746630067) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3412 1284276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3604880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3604524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7209404 1% /mnt/lustre test_70e fail mds1 2 times Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:03:30 (1746630210) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:44 (1746630224) targets are mounted 11:03:44 (1746630224) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (320s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 11:03:58 (1746630238) mount clients oleg148-client.virtnet ... Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3192 1284496 1% /mnt/lustre[MDT:0] ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3597140 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7190100 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:04:07 (1746630247) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Started lustre-OST0000 11:04:22 (1746630262) targets are mounted 11:04:22 (1746630262) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3192 1284496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3586768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3593020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7179788 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:04:37 (1746630277) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Started lustre-OST0000 11:04:53 (1746630293) targets are mounted 11:04:53 (1746630293) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3192 1284496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7724 3586816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3588828 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 9292 7175644 1% /mnt/lustre ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... test_70f failing OST 3 times Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:05:08 (1746630308) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Started lustre-OST0000 11:05:24 (1746630324) targets are mounted 11:05:24 (1746630324) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3192 1284496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 5676 3599248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 4640 3601332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 10316 7200580 1% /mnt/lustre ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... test_70f failing OST 4 times Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:05:38 (1746630338) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... Started lustre-OST0000 11:05:55 (1746630355) targets are mounted 11:05:55 (1746630355) facet_failover done ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg148-client.virtnet' ... ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a972f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a972f000.lru_size=clear PASS 70f (125s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 11:06:03 (1746630363) Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 32758 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4380 1283308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4648 1283040 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4564 1283124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2824 1284864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail mds2 mds1 1 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:08:26 (1746630506) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server Failover mds2 to oleg148-server mount facets: mds2 mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:08:41 (1746630521) targets are mounted 11:08:41 (1746630521) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2932 1284756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3848 1283840 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3104 1284584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4028 1283660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail mds2 mds1 2 times Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:11:20 (1746630680) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server Failover mds2 to oleg148-server mount facets: mds2 mount facets: mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:11:36 (1746630696) targets are mounted 11:11:36 (1746630696) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 32758 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (348s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 11:11:52 (1746630712) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3756 1283932 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3536 1284152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:11:57 (1746630717) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:12:12 (1746630732) targets are mounted 11:12:12 (1746630732) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (41s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 11:12:32 (1746630752) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7636 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1285640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:12:37 (1746630757) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:12:52 (1746630772) targets are mounted 11:12:52 (1746630772) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (39s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 11:13:11 (1746630791) Stopping clients: oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg148-server Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:13:15 (1746630795) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:13:30 (1746630810) targets are mounted 11:13:30 (1746630810) facet_failover done Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (31s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 11:13:42 (1746630822) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:13:47 (1746630827) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:14:01 (1746630841) targets are mounted 11:14:01 (1746630841) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.09 seconds: 225.51 ops/second PASS 80a (28s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 11:14:10 (1746630850) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:14:24 (1746630864) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:14:39 (1746630879) targets are mounted 11:14:39 (1746630879) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.10 seconds: 207.40 ops/second PASS 80b (37s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 11:14:47 (1746630887) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:14:55 (1746630895) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:15:10 (1746630910) targets are mounted 11:15:10 (1746630910) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:15:18 (1746630918) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:15:34 (1746630934) targets are mounted 11:15:34 (1746630934) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.09 seconds: 218.02 ops/second PASS 80c (55s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 11:15:42 (1746630942) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:16:00 (1746630960) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:16:20 (1746630980) targets are mounted 11:16:20 (1746630980) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.10 seconds: 197.82 ops/second PASS 80d (48s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 11:16:30 (1746630990) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:16:38 (1746630998) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:16:53 (1746631013) targets are mounted 11:16:53 (1746631013) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.10 seconds: 206.01 ops/second PASS 80e (31s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 11:17:01 (1746631021) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:17:07 (1746631027) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:17:22 (1746631042) targets are mounted 11:17:22 (1746631042) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.11 seconds: 189.17 ops/second PASS 80f (30s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 11:17:31 (1746631051) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:17:42 (1746631062) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:57 (1746631077) targets are mounted 11:17:57 (1746631077) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:18:05 (1746631085) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:18:20 (1746631100) targets are mounted 11:18:20 (1746631100) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.09 seconds: 214.89 ops/second PASS 80g (58s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 11:18:29 (1746631109) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:18:41 (1746631121) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:19:07 (1746631147) targets are mounted 11:19:07 (1746631147) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.07 seconds: 272.97 ops/second PASS 80h (47s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 11:19:17 (1746631157) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:19:27 (1746631167) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:19:43 (1746631183) targets are mounted 11:19:43 (1746631183) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (35s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 11:19:51 (1746631191) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2128 1285560 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:19:55 (1746631195) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:10 (1746631210) targets are mounted 11:20:10 (1746631210) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (27s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 11:20:19 (1746631219) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:20:26 (1746631226) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:42 (1746631242) targets are mounted 11:20:42 (1746631242) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:20:49 (1746631249) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:21:04 (1746631264) targets are mounted 11:21:04 (1746631264) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (54s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 11:21:13 (1746631273) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:21:29 (1746631289) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:21:48 (1746631308) targets are mounted 11:21:48 (1746631308) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (56s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 11:22:09 (1746631329) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:22:13 (1746631333) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:22:28 (1746631348) targets are mounted 11:22:28 (1746631348) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (27s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 11:22:36 (1746631356) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:22:40 (1746631360) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:22:55 (1746631375) targets are mounted 11:22:55 (1746631375) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (27s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 11:23:03 (1746631383) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:23:11 (1746631391) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:23:26 (1746631406) targets are mounted 11:23:26 (1746631406) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:23:34 (1746631414) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:23:49 (1746631429) targets are mounted 11:23:49 (1746631429) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (54s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 11:23:57 (1746631437) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3148 7210892 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:24:07 (1746631447) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:24:33 (1746631473) targets are mounted 11:24:33 (1746631473) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (46s) == replay-single test 84a: stale open during export disconnect ========================================================== 11:24:43 (1746631483) fail_loc=0x80000144 total: 1 open/close in 0.01 seconds: 137.82 ops/second pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 PASS 84a (5s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 11:24:48 (1746631488) before recovery: unused locks count = 201 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:24:52 (1746631492) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:25:06 (1746631506) targets are mounted 11:25:06 (1746631506) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (26s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 11:25:14 (1746631514) before recovery: unused locks count = 100 Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:25:21 (1746631521) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 11:25:35 (1746631535) targets are mounted 11:25:35 (1746631535) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (27s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 11:25:41 (1746631541) Stopping clients: oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) PASS 86 (13s) == replay-single test 87a: write replay ================== 11:25:54 (1746631554) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1780 3605240 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3548 7210492 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.135729 s, 61.8 MB/s Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:26:00 (1746631560) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 11:26:16 (1746631576) targets are mounted 11:26:16 (1746631576) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0813181 s, 103 MB/s PASS 87a (28s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 11:26:22 (1746631582) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9972 3597048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11740 7202300 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.156143 s, 53.7 MB/s 8+0 records in 8+0 records out 8 bytes (8 B) copied, 0.00261833 s, 3.1 kB/s Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:26:29 (1746631589) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 11:26:45 (1746631605) targets are mounted 11:26:45 (1746631605) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes (72 B) copied, 0.0056939 s, 12.6 kB/s PASS 87b (29s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 11:26:51 (1746631611) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9976 3597044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11744 7202296 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9976 3597044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11744 7202296 1% /mnt/lustre before test: last_id = 14561, next_id = 14532 Creating to objid 14561 on ost lustre-OST0000... total: 31 open/close in 0.08 seconds: 393.61 ops/second total: 8 open/close in 0.02 seconds: 423.49 ops/second before recovery: last_id = 14593, next_id = 14570 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg148-server oleg148-server.virtnet Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 14601, next_id = 14570 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0448879 s, 11.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0407806 s, 12.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0358043 s, 14.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0310541 s, 16.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0288372 s, 18.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0267553 s, 19.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0242292 s, 21.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0234053 s, 22.4 MB/s -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14532 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14533 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14534 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14535 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14536 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14537 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14538 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14539 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14540 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14541 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14542 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14543 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14544 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14545 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14546 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14547 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14548 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14549 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14550 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14551 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14552 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14553 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14554 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14555 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14556 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14557 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14558 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14559 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14560 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14561 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14562 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14563 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14564 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14565 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14566 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14567 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14568 -rw-r--r-- 1 root root 0 May 7 11:26 /mnt/lustre/d88.replay-single/f-14569 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14573 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14574 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14575 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14576 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14577 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14578 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14579 -rw-r--r-- 1 root root 524288 May 7 11:27 /mnt/lustre/d88.replay-single/f-14580 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0408351 s, 12.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0438149 s, 12.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0499668 s, 10.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0412371 s, 12.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0440904 s, 11.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0413797 s, 12.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0403038 s, 13.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0436466 s, 12.0 MB/s PASS 88 (54s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 11:27:45 (1746631665) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Waiting for MDT destroys to complete 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.116865 s, 89.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:27:52 (1746631672) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:06 (1746631686) targets are mounted 11:28:06 (1746631686) facet_failover done Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7646296 free_after: 7646296 PASS 89 (113s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 11:29:38 (1746631778) Create the files Fail ost1 lustre-OST0000_UUID, display the list of affected files Stopping /mnt/lustre-ost1 (opts:) on oleg148-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/f0 /mnt/lustre/d90.replay-single/all Querying files on shutdown ost1: lfs find --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/f0 /mnt/lustre/d90.replay-single/all Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/f0 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 14603 0x390b 0x280000401 * /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 0 14602 0x390a 0x280000401 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Failover ost1 to oleg148-server oleg148-server.virtnet Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 90 (19s) == replay-single test 93a: replay + reconnect ============ 11:29:57 (1746631797) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00290879 s, 352 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg148-server Stopping /mnt/lustre-ost1 (opts:) on oleg148-server 11:29:59 (1746631799) shut down facet: ost1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover ost1 to oleg148-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 11:30:13 (1746631813) targets are mounted 11:30:13 (1746631813) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (59s) == replay-single test 93b: replay + reconnect on mds ===== 11:30:56 (1746631856) total: 20 open/close in 0.14 seconds: 144.12 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:30:59 (1746631859) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:31:13 (1746631873) targets are mounted 11:31:13 (1746631873) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (105s) == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 11:32:41 (1746631961) fail_loc=0x1701 Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:32:43 (1746631963) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:32:58 (1746631978) targets are mounted 11:32:58 (1746631978) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000347f0:0x1:0x0] 1 [0x24000abfa:0x1:0x0] total: 20 open/close in 0.08 seconds: 240.23 ops/second PASS 100a (24s) == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 11:33:05 (1746631985) fail_loc=0x119 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:33:07 (1746631987) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:33:21 (1746632001) targets are mounted 11:33:21 (1746632001) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000347f0:0x4:0x0] 1 [0x24000abfa:0x4:0x0] total: 20 open/close in 0.10 seconds: 194.40 ops/second PASS 100b (25s) == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 11:33:30 (1746632010) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1285108 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2176 1285512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds2 (opts:) on oleg148-server Failover mds2 to oleg148-server oleg148-server.virtnet Starting mds2: -o localrecov -o abort_recov_mdt /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 total: 20 open/close in 0.11 seconds: 177.52 ops/second Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:33:53 (1746632033) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:34:05 (1746632045) targets are mounted 11:34:05 (1746632045) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 1 [0x24000b3c8:0x3:0x0] 0 [0x2000347f1:0x3:0x0] total: 20 open/close in 0.06 seconds: 355.46 ops/second PASS 100c (42s) == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 11:34:13 (1746632053) striped dir -i0 -c2 -H crush2 /mnt/lustre/d100d.replay-single total: 100 mkdir in 0.42 seconds: 235.33 ops/second lustre-MDT0000-osd [catalog]: [0x20001a210:0x1:0x0] [index]: 00020 [logid]: [0x200034fc0:0x1:0x0] lustre-MDT0001-osp-MDT0000 [catalog]: [0x2400007ec:0x1:0x0] [index]: 00032 [logid]: [0x24000b3c9:0x1:0x0] [index]: 00033 [logid]: [0x24000b3c9:0x2:0x0] Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 find: '/mnt/lustre/d100d.replay-single': No such file or directory find: '/mnt/lustre/d100d.replay-single': No such file or directory PASS 100d (21s) == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 11:34:33 (1746632073) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3000 1284688 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2592 1285096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3000 1284688 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2592 1285096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200035790:0x2:0x0] 1 [0x24000c369:0x2:0x0] Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:34:42 (1746632082) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:35:08 (1746632108) targets are mounted 11:35:08 (1746632108) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200035790:0x2:0x0] 1 [0x24000c369:0x2:0x0] PASS 100e (53s) == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 11:35:26 (1746632126) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3080 1284608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2664 1285024 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 PASS 101 (51s) == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 11:36:17 (1746632177) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (11:36:19) ... fail_loc=0 done (11:36:35) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (19s) == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 11:36:36 (1746632196) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (11:36:37) ... fail_loc=0 done (11:36:53) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (19s) == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 11:36:55 (1746632215) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3012 1284676 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3580644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7177692 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (11:36:59) ... fail_loc=0 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:37:02 (1746632222) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:37:16 (1746632236) targets are mounted 11:37:16 (1746632236) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (11:37:23) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (29s) == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 11:37:24 (1746632244) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (11:37:25) ... fail_loc=0 Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:37:28 (1746632248) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:37:41 (1746632261) targets are mounted 11:37:41 (1746632261) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec done (11:37:48) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (25s) == replay-single test 103: Check otr_next_id overflow ==== 11:37:49 (1746632269) fail_loc=0x80000162 total: 30 open/close in 0.12 seconds: 260.46 ops/second Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:37:52 (1746632272) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:38:05 (1746632285) targets are mounted 11:38:05 (1746632285) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (24s) == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 11:38:13 (1746632293) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2960 1284728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3580644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7177692 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:38:17 (1746632297) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:38:31 (1746632311) targets are mounted 11:38:31 (1746632311) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110a.replay-single/striped_dir has type dir OK PASS 110a (27s) == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 11:38:40 (1746632320) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2964 1284724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3580644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7177692 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:38:45 (1746632325) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:39:00 (1746632340) targets are mounted 11:39:00 (1746632340) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre /mnt/lustre/d110b.replay-single/striped_dir has type dir OK PASS 110b (98s) == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 11:40:18 (1746632418) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2964 1284724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:40:23 (1746632423) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:40:38 (1746632438) targets are mounted 11:40:38 (1746632438) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110c.replay-single/striped_dir has type dir OK PASS 110c (28s) == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 11:40:46 (1746632446) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3008 1284680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:40:51 (1746632451) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:41:07 (1746632467) targets are mounted 11:41:07 (1746632467) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre /mnt/lustre/d110d.replay-single/striped_dir has type dir OK PASS 110d (97s) == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 11:42:23 (1746632543) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2956 1284732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:42:31 (1746632551) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:42:56 (1746632576) targets are mounted 11:42:56 (1746632576) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre /mnt/lustre/d110e.replay-single/striped_dir has type dir OK PASS 110e (108s) SKIP: replay-single test_110f skipping excluded test 110f == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:44:12 (1746632652) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2960 1284728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:44:21 (1746632661) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:44:47 (1746632687) targets are mounted 11:44:47 (1746632687) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre /mnt/lustre/d110g.replay-single/striped_dir has type dir OK PASS 110g (111s) == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 11:46:03 (1746632763) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3008 1284680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2532 1285156 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:46:08 (1746632768) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:46:24 (1746632784) targets are mounted 11:46:24 (1746632784) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111a.replay-single/striped_dir: No such file or directory PASS 111a (29s) == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 11:46:32 (1746632792) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:46:37 (1746632797) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:46:52 (1746632812) targets are mounted 11:46:52 (1746632812) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111b.replay-single/striped_dir: No such file or directory PASS 111b (96s) == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:48:08 (1746632888) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2972 1284716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2536 1285152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:48:23 (1746632903) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:48:43 (1746632923) targets are mounted 11:48:43 (1746632923) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111c.replay-single/striped_dir: No such file or directory PASS 111c (111s) == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 11:49:59 (1746632999) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2980 1284708 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2536 1285152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:50:07 (1746633007) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:50:33 (1746633033) targets are mounted 11:50:33 (1746633033) facet_failover done pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg148-client: oleg148-client: ssh exited with exit code 95 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111d.replay-single/striped_dir: No such file or directory PASS 111d (110s) == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 11:51:49 (1746633109) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2536 1285152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:51:58 (1746633118) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:52:24 (1746633144) targets are mounted 11:52:24 (1746633144) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111e.replay-single/striped_dir: No such file or directory PASS 111e (45s) == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 11:52:34 (1746633154) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2984 1284704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2984 1284704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2532 1285156 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:52:44 (1746633164) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:53:09 (1746633189) targets are mounted 11:53:09 (1746633189) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111f.replay-single/striped_dir: No such file or directory PASS 111f (40s) == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 11:53:14 (1746633194) UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 552 1023448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 322 1023678 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 557 261587 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 486 261658 1% /mnt/lustre[OST:1] filesystem_summary: 524119 874 523245 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2980 1284708 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2532 1285156 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2992 1284696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:53:20 (1746633200) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:53:45 (1746633225) targets are mounted 11:53:45 (1746633225) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111g.replay-single/striped_dir: No such file or directory PASS 111g (41s) == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 11:53:55 (1746633235) SKIP: replay-single test_112a needs >= 4 MDTs SKIP 112a (1s) == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 11:53:56 (1746633236) SKIP: replay-single test_112b needs >= 4 MDTs SKIP 112b (1s) == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 11:53:58 (1746633238) SKIP: replay-single test_112c needs >= 4 MDTs SKIP 112c (1s) == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 11:53:59 (1746633239) SKIP: replay-single test_112d needs >= 4 MDTs SKIP 112d (1s) == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 11:54:00 (1746633240) SKIP: replay-single test_112e needs >= 4 MDTs SKIP 112e (1s) == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 11:54:01 (1746633241) SKIP: replay-single test_112f needs >= 4 MDTs SKIP 112f (1s) == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 11:54:02 (1746633242) SKIP: replay-single test_112g needs >= 4 MDTs SKIP 112g (2s) == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 11:54:04 (1746633244) SKIP: replay-single test_112h needs >= 4 MDTs SKIP 112h (1s) == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 11:54:05 (1746633245) SKIP: replay-single test_112i needs >= 4 MDTs SKIP 112i (1s) == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 11:54:06 (1746633246) SKIP: replay-single test_112j needs >= 4 MDTs SKIP 112j (1s) == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 11:54:07 (1746633247) SKIP: replay-single test_112k needs >= 4 MDTs SKIP 112k (1s) == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 11:54:08 (1746633248) SKIP: replay-single test_112l needs >= 4 MDTs SKIP 112l (1s) == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 11:54:09 (1746633249) SKIP: replay-single test_112m needs >= 4 MDTs SKIP 112m (1s) == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 11:54:10 (1746633250) SKIP: replay-single test_112n needs >= 4 MDTs SKIP 112n (2s) == replay-single test 115: failover for create/unlink striped directory ========================================================== 11:54:12 (1746633252) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2988 1284700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre striped dir -i1 -c2 -H crush /mnt/lustre/d115.replay-single/test_0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_2 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_3 striped dir -i1 -c2 -H all_char /mnt/lustre/d115.replay-single/test_4 Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:54:17 (1746633257) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:54:32 (1746633272) targets are mounted 11:54:32 (1746633272) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3040 1284648 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2596 1285092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre striped dir -i0 -c2 -H all_char /mnt/lustre/d115.replay-single/test_0 striped dir -i0 -c2 -H all_char /mnt/lustre/d115.replay-single/test_1 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_2 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_3 striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_4 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:54:43 (1746633283) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:54:58 (1746633298) targets are mounted 11:54:58 (1746633298) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 115 (55s) == replay-single test 116a: large update log master MDT recovery ========================================================== 11:55:07 (1746633307) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3080 1284608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2660 1285028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre fail_loc=0x80001702 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:55:12 (1746633312) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:55:27 (1746633327) targets are mounted 11:55:27 (1746633327) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116a.replay-single/striped_dir has type dir OK PASS 116a (27s) == replay-single test 116b: large update log slave MDT recovery ========================================================== 11:55:34 (1746633334) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3132 1284556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2832 1284856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre fail_loc=0x80001702 Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 11:55:39 (1746633339) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 11:55:54 (1746633354) targets are mounted 11:55:54 (1746633354) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116b.replay-single/striped_dir has type dir OK PASS 116b (28s) == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 11:56:02 (1746633362) SKIP: replay-single test_117 needs >= 4 MDTs SKIP 117 (1s) == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 11:56:03 (1746633363) fail_loc=0x1705 lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error fail_loc=0x0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3228 1284460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2712 1284976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 11:56:07 (1746633367) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 11:56:21 (1746633381) targets are mounted 11:56:21 (1746633381) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 118 (25s) == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 11:56:28 (1746633388) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3296 1284392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2772 1284916 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet fail_loc=0x80000714 fail_val=65 Starting mds1: -o localrecov -o recovery_time_hard=60 /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 70 sec mdt.lustre-MDT0000.recovery_time_hard=180 PASS 119 (85s) == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 11:57:53 (1746633473) Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 first stat failed: 5 find: '/mnt/lustre/d120.replay-single': No such file or directory find: '/mnt/lustre/d120.replay-single': No such file or directory PASS 120 (19s) == replay-single test 121: lock replay timed out and race ========================================================== 11:58:12 (1746633492) multiop /mnt/lustre/f121.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7636 Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Failover mds1 to oleg148-server oleg148-server.virtnet fail_loc=0x721 fail_val=0 at_max=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 fail_loc=0x0 at_max=600 PASS 121 (198s) == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 12:01:30 (1746633690) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1283992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3076 1284612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:01:33 (1746633693) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:01:47 (1746633707) targets are mounted 12:01:47 (1746633707) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130a (24s) == replay-single test 130b: DoM file create (inherited) replay ========================================================== 12:01:54 (1746633714) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1283992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3076 1284612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:01:57 (1746633717) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:02:11 (1746633731) targets are mounted 12:02:11 (1746633731) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130b (24s) == replay-single test 131a: DoM file write lock replay === 12:02:18 (1746633738) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3700 1283988 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3076 1284612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre 1+0 records in 1+0 records out 8 bytes (8 B) copied, 0.00136923 s, 5.8 kB/s Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:02:22 (1746633742) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:02:35 (1746633755) targets are mounted 12:02:35 (1746633755) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 131a (25s) SKIP: replay-single test_131b skipping excluded test 131b == replay-single test 132a: PFL new component instantiate replay ========================================================== 12:02:43 (1746633763) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3704 1283984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3076 1284612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18172 3588848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19940 7194100 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159958 s, 65.6 MB/s /mnt/lustre/f132a.replay-single lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4002:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x404a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4003:0x0] } Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:02:46 (1746633766) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:03:00 (1746633780) targets are mounted 12:03:00 (1746633780) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f132a.replay-single lcm_layout_gen: 4 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4002:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x404a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4003:0x0] } PASS 132a (25s) == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 12:03:08 (1746633788) seq.srv-lustre-MDT0001.space=clear Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre fail_val=700 fail_loc=0x80000123 Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:03:12 (1746633792) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:03:25 (1746633805) targets are mounted 12:03:25 (1746633805) facet_failover done PASS 133 (22s) == replay-single test 134: replay creation of a file created in a pool ========================================================== 12:03:30 (1746633810) Creating new pool pool_134 oleg148-server: Pool lustre.pool_134 created Adding targets to pool oleg148-server: OST lustre-OST0001_UUID added to pool lustre.pool_134 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3784 1283904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3120 1284568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20964 7193076 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:03:37 (1746633817) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:03:51 (1746633831) targets are mounted 12:03:51 (1746633831) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: pool_134 lustre.pool_134 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.pool_134 oleg148-server: Pool lustre.pool_134 destroyed PASS 134 (33s) == replay-single test 135: Server failure in lock replay phase ========================================================== 12:04:03 (1746633843) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3764 1283924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3120 1284568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20964 7193076 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg148-server Failover ost1 to oleg148-server oleg148-server.virtnet oleg148-server: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg148-server Failover ost1 to oleg148-server oleg148-server.virtnet oleg148-server: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all End of sync pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg148-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 5 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135 (63s) == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 12:05:06 (1746633906) SKIP: replay-single test_136 needs > 2 MDTs SKIP 136 (1s) == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 12:05:07 (1746633907) llite.lustre-ffff88012b19f000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3772 1283916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3140 1284548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19272 3587748 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 21044 7192996 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:05:10 (1746633910) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:05:24 (1746633924) targets are mounted 12:05:24 (1746633924) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137a.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137a.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137a (24s) == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 12:05:31 (1746633931) llite.lustre-ffff88012b19f000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3780 1283908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3144 1284544 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 12:05:35 (1746633935) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds2 to oleg148-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 12:05:49 (1746633949) targets are mounted 12:05:49 (1746633949) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137b.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137b.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137b (25s) == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 12:05:56 (1746633956) llite.lustre-ffff88012b19f000.intent_mkdir=1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3776 1283912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3132 1284556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3776 1283912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3132 1284556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds2 on oleg148-server Stopping /mnt/lustre-mds2 (opts:) on oleg148-server Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:06:03 (1746633963) shut down facet: mds2 facet_host: oleg148-server facet_failover_host: oleg148-server facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server Failover mds2 to oleg148-server mount facets: mds2 mount facets: mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 12:06:17 (1746633977) targets are mounted 12:06:17 (1746633977) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d137c.replay-single/striped_dir/dir0 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] /mnt/lustre/d137c.replay-single/striped_dir/dir1 has type dir OK lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 137c (30s) == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 12:06:26 (1746633986) SKIP: replay-single test_200 Need remote client SKIP 200 (1s) == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 12:06:27 (1746633987) fail_loc=0x245 fail_val=8 fail_loc=0x245 fail_val=8 Stopping /mnt/lustre-mds2 (opts:) on oleg148-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Umount took 17 seconds PASS 201 (23s) == replay-single test 202: pfl replay should recovery layout generation ========================================================== 12:06:50 (1746634010) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3804 1283884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3084 1284604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19196 3587824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1772 3605248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 12:06:55 (1746634015) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 12:07:09 (1746634029) targets are mounted 12:07:09 (1746634029) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 202 (26s) == replay-single test complete, duration 8481 sec ======== 12:07:16 (1746634036) === replay-single: start cleanup 12:07:17 (1746634037) === === replay-single: finish cleanup 12:07:20 (1746634040) ===