-----============= acceptance-small: replay-single ============----- Wed Apr 22 10:54:29 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg147-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 36 === replay-single: start setup 10:54:36 (1776869676) === oleg147-client.virtnet: executing check_config_client /mnt/lustre oleg147-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg147-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9daed04be000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9daed04be000.idle_timeout=debug disable quota as required oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 10:54:49 (1776869689) === == replay-single test 0a: empty replay =================== 10:54:50 (1776869690) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1712 1269592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 10:54:56 (1776869696) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 10:55:13 (1776869713) targets are mounted 10:55:13 (1776869713) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (32s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 10:55:22 (1776869722) Failing ost1 on oleg147-server Stopping /mnt/lustre-ost1 (opts:) on oleg147-server 10:55:24 (1776869724) shut down facet: ost1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover ost1 to oleg147-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 10:55:40 (1776869740) targets are mounted 10:55:40 (1776869740) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.14 seconds: 141.51 ops/second - unlinked 0 (time 1776869745 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (25s) == replay-single test 0c: check replay-barrier =========== 10:55:47 (1776869747) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 10:55:53 (1776869753) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 10:56:09 (1776869769) targets are mounted 10:56:09 (1776869769) facet_failover done Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (98s) == replay-single test 0d: expired recovery with no clients ========================================================== 10:57:25 (1776869845) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 10:57:31 (1776869851) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 10:57:48 (1776869868) targets are mounted 10:57:48 (1776869868) facet_failover done Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre PASS 0d (98s) == replay-single test 1: simple create =================== 10:59:03 (1776869943) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 10:59:08 (1776869948) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 10:59:24 (1776869964) targets are mounted 10:59:24 (1776869964) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (29s) == replay-single test 2a: touch ========================== 10:59:32 (1776869972) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 10:59:37 (1776869977) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 10:59:53 (1776869993) targets are mounted 10:59:53 (1776869993) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (29s) == replay-single test 2b: touch ========================== 11:00:02 (1776870002) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:00:06 (1776870006) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:22 (1776870022) targets are mounted 11:00:22 (1776870022) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (30s) == replay-single test 2c: setstripe replay =============== 11:00:31 (1776870031) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:00:36 (1776870036) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:52 (1776870052) targets are mounted 11:00:52 (1776870052) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (29s) == replay-single test 2d: setdirstripe replay ============ 11:01:00 (1776870060) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:01:05 (1776870065) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:01:21 (1776870081) targets are mounted 11:01:21 (1776870081) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (29s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:01:29 (1776870089) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:01:35 (1776870095) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:01:51 (1776870111) targets are mounted 11:01:51 (1776870111) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (30s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:01:59 (1776870119) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:02:04 (1776870124) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:02:20 (1776870140) targets are mounted 11:02:20 (1776870140) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (29s) == replay-single test 3b: replay failed open -ENOMEM ===== 11:02:28 (1776870148) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:02:34 (1776870154) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:02:51 (1776870171) targets are mounted 11:02:51 (1776870171) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (31s) == replay-single test 3c: replay failed open -ENOMEM ===== 11:02:59 (1776870179) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:03:04 (1776870184) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:21 (1776870201) targets are mounted 11:03:21 (1776870201) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (30s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:03:29 (1776870209) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:03:33 (1776870213) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:48 (1776870228) targets are mounted 11:03:48 (1776870228) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (28s) == replay-single test 4b: |x| rm 10 files ================ 11:03:57 (1776870237) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:04:01 (1776870241) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:04:16 (1776870256) targets are mounted 11:04:16 (1776870256) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (27s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:04:24 (1776870264) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1596 3605424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:04:30 (1776870270) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:04:46 (1776870286) targets are mounted 11:04:46 (1776870286) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (36s) == replay-single test 6a: mkdir + contained create ======= 11:05:00 (1776870300) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2016 3605004 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2000 3605020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4016 7210024 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:05:04 (1776870304) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:05:19 (1776870319) targets are mounted 11:05:19 (1776870319) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (29s) == replay-single test 6b: |X| rmdir ====================== 11:05:29 (1776870329) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:05:34 (1776870334) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:05:49 (1776870349) targets are mounted 11:05:49 (1776870349) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (28s) == replay-single test 7: mkdir |X| contained create ====== 11:05:57 (1776870357) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:06:01 (1776870361) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:06:17 (1776870377) targets are mounted 11:06:17 (1776870377) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (27s) == replay-single test 8: creat open |X| close ============ 11:06:24 (1776870384) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:06:29 (1776870389) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:06:44 (1776870404) targets are mounted 11:06:44 (1776870404) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (28s) == replay-single test 9: |X| create (same inum/gen) ====== 11:06:52 (1776870412) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:06:56 (1776870416) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:07:11 (1776870431) targets are mounted 11:07:11 (1776870431) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (27s) == replay-single test 10: create |X| rename unlink ======= 11:07:19 (1776870439) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:07:23 (1776870443) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:07:38 (1776870458) targets are mounted 11:07:38 (1776870458) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (27s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:07:46 (1776870466) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre new old Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:07:51 (1776870471) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:08:06 (1776870486) targets are mounted 11:08:06 (1776870486) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (28s) == replay-single test 12: open, unlink |X| close ========= 11:08:14 (1776870494) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:08:18 (1776870498) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:08:34 (1776870514) targets are mounted 11:08:34 (1776870514) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (27s) == replay-single test 13: open chmod 0 |x| write close === 11:08:41 (1776870521) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:08:45 (1776870525) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:09:01 (1776870541) targets are mounted 11:09:01 (1776870541) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (28s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:09:09 (1776870549) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:09:13 (1776870553) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:09:28 (1776870568) targets are mounted 11:09:28 (1776870568) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (27s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:09:36 (1776870576) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:09:40 (1776870580) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:09:55 (1776870595) targets are mounted 11:09:55 (1776870595) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (27s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:10:03 (1776870603) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:10:07 (1776870607) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:10:23 (1776870623) targets are mounted 11:10:23 (1776870623) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (28s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:10:31 (1776870631) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:10:35 (1776870635) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:10:50 (1776870650) targets are mounted 11:10:50 (1776870650) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (27s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:10:58 (1776870658) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 pid: 52471 will close Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:11:02 (1776870662) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:11:17 (1776870677) targets are mounted 11:11:17 (1776870677) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (27s) == replay-single test 19: mcreate, open, write, rename === 11:11:25 (1776870685) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre old Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:11:30 (1776870690) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:11:45 (1776870705) targets are mounted 11:11:45 (1776870705) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (28s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:11:53 (1776870713) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3156 7210884 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:11:57 (1776870717) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:12:13 (1776870733) targets are mounted 11:12:13 (1776870733) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (27s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:12:20 (1776870740) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x280000401 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown pdsh@oleg147-client: oleg147-client: ssh exited with exit code 5 7130+0 records in 7129+0 records out 29200384 bytes (29 MB, 28 MiB) copied, 1.65452 s, 17.6 MB/s Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:12:27 (1776870747) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:12:41 (1776870761) targets are mounted 11:12:41 (1776870761) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg147-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 3156, after 3156 PASS 20b (33s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:12:54 (1776870774) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 -rw-r--r-- 1 root root 1 Apr 22 11:12 /mnt/lustre/f20c.replay-single PASS 20c (4s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:12:57 (1776870777) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3156 7210852 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:13:02 (1776870782) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:13:18 (1776870798) targets are mounted 11:13:18 (1776870798) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (29s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:13:26 (1776870806) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:13:31 (1776870811) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:13:47 (1776870827) targets are mounted 11:13:47 (1776870827) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (29s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:13:55 (1776870835) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:14:00 (1776870840) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:14:15 (1776870855) targets are mounted 11:14:15 (1776870855) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (28s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:14:24 (1776870864) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:14:28 (1776870868) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:14:43 (1776870883) targets are mounted 11:14:43 (1776870883) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (27s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:14:51 (1776870891) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:14:56 (1776870896) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:15:11 (1776870911) targets are mounted 11:15:11 (1776870911) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (28s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:15:19 (1776870919) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:15:24 (1776870924) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:15:40 (1776870940) targets are mounted 11:15:40 (1776870940) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (28s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:15:48 (1776870948) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:15:52 (1776870952) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:16:08 (1776870968) targets are mounted 11:16:08 (1776870968) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (29s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:16:16 (1776870976) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:16:20 (1776870980) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:16:36 (1776870996) targets are mounted 11:16:36 (1776870996) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (28s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:16:44 (1776871004) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:16:48 (1776871008) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:04 (1776871024) targets are mounted 11:17:04 (1776871024) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (28s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:17:12 (1776871032) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:17:16 (1776871036) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:31 (1776871051) targets are mounted 11:17:31 (1776871051) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (27s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:17:39 (1776871059) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1608 1269696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:17:43 (1776871063) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:59 (1776871079) targets are mounted 11:17:59 (1776871079) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (27s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:18:07 (1776871087) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 5 PASS 32 (6s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:18:12 (1776871092) total: 10 open/close in 0.05 seconds: 208.40 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg147-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg147-client: oleg147-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.05 seconds: 194.29 ops/second PASS 33a (20s) == replay-single test 33b: test fid seq allocation ======= 11:18:32 (1776871112) fail_loc=0x1311 total: 10 open/close in 0.04 seconds: 240.46 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.05 seconds: 213.50 ops/second PASS 33b (21s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:18:53 (1776871133) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1816 1269488 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1640 1269664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (20s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:19:13 (1776871153) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg147-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg147-client: oleg147-client: ssh exited with exit code 5 first stat failed: 5 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (20s) SKIP: replay-single test_36 skipping ALWAYS excluded test 36 == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:19:34 (1776871174) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1269416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1704 1269600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (20s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:19:54 (1776871194) total: 800 open/close in 2.39 seconds: 334.45 ops/second - unlinked 0 (time 1776871199 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:20:04 (1776871204) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:20 (1776871220) targets are mounted 11:20:20 (1776871220) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776871227 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (38s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:20:32 (1776871232) total: 800 open/close in 2.66 seconds: 301.24 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210848 1% /mnt/lustre - unlinked 0 (time 1776871240 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:20:43 (1776871243) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:58 (1776871258) targets are mounted 11:20:58 (1776871258) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776871266 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (39s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:21:11 (1776871271) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00198144 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00384333 s, 1.1 MB/s PASS 41 (3s) == replay-single test 42: recovery after ost failure ===== 11:21:14 (1776871274) total: 800 open/close in 2.43 seconds: 328.84 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre - unlinked 0 (time 1776871282 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second debug=-1 Failing ost1 on oleg147-server Stopping /mnt/lustre-ost1 (opts:) on oleg147-server 11:21:25 (1776871285) shut down facet: ost1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover ost1 to oleg147-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 11:21:41 (1776871301) targets are mounted 11:21:41 (1776871301) facet_failover done wait for MDS to timeout and recover - unlinked 0 (time 1776871343 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (71s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:22:25 (1776871345) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2096 1269208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:22:30 (1776871350) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:22:46 (1776871366) targets are mounted 11:22:46 (1776871366) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (39s) == replay-single test 44a: race in target handle connect ========================================================== 11:23:04 (1776871384) at_max=40 1 of 10 (1776871385) service : cur 5 worst 5 (at 1776869633, 1753s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 2 of 10 (1776871391) service : cur 5 worst 5 (at 1776869633, 1758s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 3 of 10 (1776871397) service : cur 5 worst 5 (at 1776869633, 1764s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 4 of 10 (1776871402) service : cur 5 worst 5 (at 1776869633, 1770s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 5 of 10 (1776871408) service : cur 5 worst 5 (at 1776869633, 1775s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 6 of 10 (1776871414) service : cur 5 worst 5 (at 1776869633, 1782s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 7 of 10 (1776871420) service : cur 5 worst 5 (at 1776869633, 1788s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 8 of 10 (1776871426) service : cur 5 worst 5 (at 1776869633, 1793s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 9 of 10 (1776871432) service : cur 6 worst 6 (at 1776871432, 0s ago) 6 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 10 of 10 (1776871437) service : cur 6 worst 6 (at 1776871432, 5s ago) 6 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (61s) == replay-single test 44b: race in target handle connect ========================================================== 11:24:05 (1776871445) 1 of 10 (1776871446) service : cur 6 worst 6 (at 1776871432, 14s ago) 6 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 2 of 10 (1776871466) service : cur 6 worst 6 (at 1776871432, 35s ago) 6 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 3 of 10 (1776871487) service : cur 40 worst 40 (at 1776871486, 2s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 4 of 10 (1776871508) service : cur 40 worst 40 (at 1776871486, 23s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 5 of 10 (1776871529) service : cur 40 worst 40 (at 1776871486, 44s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 6 of 10 (1776871550) service : cur 40 worst 40 (at 1776871486, 65s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 7 of 10 (1776871571) service : cur 40 worst 40 (at 1776871486, 86s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 8 of 10 (1776871592) service : cur 40 worst 40 (at 1776871486, 107s ago) 1 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 9 of 10 (1776871613) service : cur 40 worst 40 (at 1776871486, 128s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 10 of 10 (1776871634) service : cur 40 worst 40 (at 1776871486, 149s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre PASS 44b (212s) == replay-single test 44c: race in target handle connect ========================================================== 11:27:37 (1776871657) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre total: 100 create in 0.25 seconds: 398.26 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg147-server Failover mds1 to oleg147-server oleg147-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:27:58 (1776871678) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:12 (1776871692) targets are mounted 11:28:12 (1776871692) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (43s) == replay-single test 45: Handle failed close ============ 11:28:20 (1776871700) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8147 /mnt/lustre/f45.replay-single has type file OK PASS 45 (3s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:28:24 (1776871704) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:28:42 (1776871722) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:56 (1776871736) targets are mounted 11:28:56 (1776871736) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (41s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:29:05 (1776871745) total: 20 open/close in 0.08 seconds: 250.50 ops/second Failing ost1 on oleg147-server Stopping /mnt/lustre-ost1 (opts:) on oleg147-server 11:29:07 (1776871747) shut down facet: ost1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover ost1 to oleg147-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 11:29:22 (1776871762) targets are mounted 11:29:22 (1776871762) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.09 seconds: 229.17 ops/second - unlinked 0 (time 1776871828 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (84s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:30:29 (1776871829) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1768 1269536 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre total: 20 open/close in 0.08 seconds: 253.72 ops/second Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 11:30:34 (1776871834) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 11:30:49 (1776871849) targets are mounted 11:30:49 (1776871849) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.11 seconds: 174.18 ops/second - unlinked 0 (time 1776871913 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (86s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 11:31:55 (1776871915) PASS 50 (8s)