-----============= acceptance-small: replay-single ============----- Wed Sep 9 12:08:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg355-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 36 === replay-single: start setup 12:09:11 (1788970151) === oleg355-client.virtnet: executing check_config_client /mnt/lustre oleg355-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg355-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ca1117ea000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ca1117ea000.idle_timeout=debug disable quota as required oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 12:09:50 (1788970190) === == replay-single test 0a: empty replay =================== 12:09:52 (1788970192) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1712 1269592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:10:06 (1788970206) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:10:41 (1788970241) targets are mounted 12:10:41 (1788970241) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (64s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 12:10:56 (1788970256) Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 12:11:02 (1788970262) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 12:11:26 (1788970286) targets are mounted 12:11:26 (1788970286) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.57 seconds: 34.83 ops/second - unlinked 0 (time 1788970298 ; total 0 ; last 0) total: 20 unlinks in 1 seconds: 20.000000 unlinks/second PASS 0b (48s) == replay-single test 0c: check replay-barrier =========== 12:11:44 (1788970304) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:12:00 (1788970320) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:12:31 (1788970351) targets are mounted 12:12:31 (1788970351) facet_failover done Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (127s) == replay-single test 0d: expired recovery with no clients ========================================================== 12:13:51 (1788970431) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:14:04 (1788970444) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:14:31 (1788970471) targets are mounted 12:14:31 (1788970471) facet_failover done Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre PASS 0d (119s) == replay-single test 1: simple create =================== 12:15:50 (1788970550) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:16:03 (1788970563) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:16:34 (1788970594) targets are mounted 12:16:34 (1788970594) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (58s) == replay-single test 2a: touch ========================== 12:16:48 (1788970608) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:17:00 (1788970620) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:17:25 (1788970645) targets are mounted 12:17:25 (1788970645) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (52s) == replay-single test 2b: touch ========================== 12:17:40 (1788970660) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:17:51 (1788970671) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:18:26 (1788970706) targets are mounted 12:18:26 (1788970706) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (62s) == replay-single test 2c: setstripe replay =============== 12:18:42 (1788970722) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:18:54 (1788970734) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:19:30 (1788970770) targets are mounted 12:19:30 (1788970770) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (64s) == replay-single test 2d: setdirstripe replay ============ 12:19:46 (1788970786) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:20:02 (1788970802) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:20:32 (1788970832) targets are mounted 12:20:32 (1788970832) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (63s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 12:20:49 (1788970849) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:21:05 (1788970865) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) Started lustre-MDT0000 12:21:39 (1788970899) targets are mounted 12:21:39 (1788970899) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (67s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 12:21:56 (1788970916) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:22:09 (1788970929) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:22:41 (1788970961) targets are mounted 12:22:41 (1788970961) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (62s) == replay-single test 3b: replay failed open -ENOMEM ===== 12:22:58 (1788970978) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:23:13 (1788970993) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:23:49 (1788971029) targets are mounted 12:23:49 (1788971029) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (69s) == replay-single test 3c: replay failed open -ENOMEM ===== 12:24:07 (1788971047) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:24:24 (1788971064) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:24:55 (1788971095) targets are mounted 12:24:55 (1788971095) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (63s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 12:25:10 (1788971110) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:25:22 (1788971122) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:25:45 (1788971145) targets are mounted 12:25:45 (1788971145) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (54s) == replay-single test 4b: |x| rm 10 files ================ 12:26:04 (1788971164) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:26:18 (1788971178) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:26:50 (1788971210) targets are mounted 12:26:50 (1788971210) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (64s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 12:27:08 (1788971228) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1728 1269576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:27:28 (1788971248) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:28:03 (1788971283) targets are mounted 12:28:03 (1788971283) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (87s) == replay-single test 6a: mkdir + contained create ======= 12:28:35 (1788971315) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1664 3605356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3228 7210812 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:28:46 (1788971326) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:29:19 (1788971359) targets are mounted 12:29:19 (1788971359) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (61s) == replay-single test 6b: |X| rmdir ====================== 12:29:36 (1788971376) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:29:50 (1788971390) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:30:22 (1788971422) targets are mounted 12:30:22 (1788971422) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (60s) == replay-single test 7: mkdir |X| contained create ====== 12:30:36 (1788971436) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:30:48 (1788971448) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:31:20 (1788971480) targets are mounted 12:31:20 (1788971480) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (58s) == replay-single test 8: creat open |X| close ============ 12:31:34 (1788971494) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:31:45 (1788971505) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:32:16 (1788971536) targets are mounted 12:32:16 (1788971536) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (58s) == replay-single test 9: |X| create (same inum/gen) ====== 12:32:32 (1788971552) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:32:44 (1788971564) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:33:18 (1788971598) targets are mounted 12:33:18 (1788971598) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (61s) == replay-single test 10: create |X| rename unlink ======= 12:33:33 (1788971613) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:33:45 (1788971625) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:34:20 (1788971660) targets are mounted 12:34:20 (1788971660) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (62s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 12:34:35 (1788971675) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre new old Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:34:48 (1788971688) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:35:21 (1788971721) targets are mounted 12:35:21 (1788971721) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (62s) == replay-single test 12: open, unlink |X| close ========= 12:35:37 (1788971737) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:35:49 (1788971749) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:36:26 (1788971786) targets are mounted 12:36:26 (1788971786) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (63s) == replay-single test 13: open chmod 0 |x| write close === 12:36:41 (1788971801) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:36:52 (1788971812) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:37:30 (1788971850) targets are mounted 12:37:30 (1788971850) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (66s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 12:37:47 (1788971867) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:38:00 (1788971880) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:38:35 (1788971915) targets are mounted 12:38:35 (1788971915) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (62s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 12:38:49 (1788971929) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:39:02 (1788971942) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:39:25 (1788971965) targets are mounted 12:39:25 (1788971965) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (54s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 12:39:43 (1788971983) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:39:55 (1788971995) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:40:28 (1788972028) targets are mounted 12:40:28 (1788972028) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (60s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 12:40:43 (1788972043) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:40:54 (1788972054) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:41:26 (1788972086) targets are mounted 12:41:26 (1788972086) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (58s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 12:41:41 (1788972101) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 pid: 53291 will close Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:41:53 (1788972113) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:42:29 (1788972149) targets are mounted 12:42:29 (1788972149) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (63s) == replay-single test 19: mcreate, open, write, rename === 12:42:44 (1788972164) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3152 7210888 1% /mnt/lustre old Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:42:57 (1788972177) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:43:30 (1788972210) targets are mounted 12:43:30 (1788972210) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (60s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:43:45 (1788972225) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3156 7210884 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:43:56 (1788972236) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:44:29 (1788972269) targets are mounted 12:44:29 (1788972269) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (61s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 12:44:46 (1788972286) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x280000401 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown 5943+0 records in 5942+0 records out 24338432 bytes (24 MB, 23 MiB) copied, 2.89222 s, 8.4 MB/s Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:44:59 (1788972299) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:45:26 (1788972326) targets are mounted 12:45:26 (1788972326) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg355-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 3156, after 3156 PASS 20b (73s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 12:45:59 (1788972359) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 -rw-r--r-- 1 root root 1 Sep 9 12:46 /mnt/lustre/f20c.replay-single pdsh@oleg355-client: oleg355-client: ssh exited with exit code 5 PASS 20c (12s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:46:12 (1788972372) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:46:24 (1788972384) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:46:56 (1788972416) targets are mounted 12:46:56 (1788972416) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (63s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:47:14 (1788972434) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:47:28 (1788972448) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:48:03 (1788972483) targets are mounted 12:48:03 (1788972483) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (65s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:48:19 (1788972499) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:48:32 (1788972512) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:49:06 (1788972546) targets are mounted 12:49:06 (1788972546) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (65s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 12:49:24 (1788972564) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:49:39 (1788972579) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:50:13 (1788972613) targets are mounted 12:50:13 (1788972613) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (66s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:50:30 (1788972630) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:50:43 (1788972643) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:51:15 (1788972675) targets are mounted 12:51:15 (1788972675) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (61s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:51:31 (1788972691) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:51:48 (1788972708) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:52:27 (1788972747) targets are mounted 12:52:27 (1788972747) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (78s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:52:49 (1788972769) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:53:03 (1788972783) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:53:39 (1788972819) targets are mounted 12:53:39 (1788972819) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (72s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:54:01 (1788972841) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:54:21 (1788972861) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:54:58 (1788972898) targets are mounted 12:54:58 (1788972898) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (79s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:55:20 (1788972920) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:55:34 (1788972934) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:56:10 (1788972970) targets are mounted 12:56:10 (1788972970) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (72s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:56:33 (1788972993) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:56:50 (1788973010) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:57:23 (1788973043) targets are mounted 12:57:24 (1788973044) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (70s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:57:43 (1788973063) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1572 1269732 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 12:57:57 (1788973077) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 12:58:31 (1788973111) targets are mounted 12:58:31 (1788973111) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (66s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:58:50 (1788973130) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 5 PASS 32 (12s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:59:02 (1788973142) total: 10 open/close in 0.30 seconds: 33.51 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.32 seconds: 31.09 ops/second PASS 33a (51s) == replay-single test 33b: test fid seq allocation ======= 12:59:54 (1788973194) fail_loc=0x1311 total: 10 open/close in 0.42 seconds: 23.67 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.32 seconds: 31.64 ops/second PASS 33b (70s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 13:01:05 (1788973265) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1604 1269700 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (63s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 13:02:07 (1788973327) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (46s) SKIP: replay-single test_36 skipping ALWAYS excluded test 36 == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 13:02:56 (1788973376) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1269448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (65s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 13:04:00 (1788973440) - open/close 491 (time 1788973455.67 total 10.00 last 49.08) total: 800 open/close in 17.10 seconds: 46.80 ops/second - unlinked 0 (time 1788973468 ; total 0 ; last 0) total: 400 unlinks in 6 seconds: 66.666664 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2004 1269300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:04:48 (1788973488) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:05:26 (1788973526) targets are mounted 13:05:26 (1788973526) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788973543 ; total 0 ; last 0) total: 400 unlinks in 9 seconds: 44.444443 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (122s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 13:06:02 (1788973562) - open/close 496 (time 1788973576.90 total 10.05 last 49.37) total: 800 open/close in 16.84 seconds: 47.52 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2004 1269300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3160 7210880 1% /mnt/lustre - unlinked 0 (time 1788973597 ; total 0 ; last 0) total: 400 unlinks in 7 seconds: 57.142857 unlinks/second Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:06:51 (1788973611) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:07:24 (1788973644) targets are mounted 13:07:24 (1788973644) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788973659 ; total 0 ; last 0) total: 400 unlinks in 11 seconds: 36.363636 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (120s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 13:08:02 (1788973682) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00426553 s, 960 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0545962 s, 75.0 kB/s PASS 41 (11s) == replay-single test 42: recovery after ost failure ===== 13:08:14 (1788973694) - open/close 467 (time 1788973708.36 total 10.01 last 46.64) total: 800 open/close in 16.68 seconds: 47.96 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre - unlinked 0 (time 1788973728 ; total 0 ; last 0) total: 400 unlinks in 6 seconds: 66.666664 unlinks/second debug=-1 Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 13:09:01 (1788973741) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 13:09:32 (1788973772) targets are mounted 13:09:32 (1788973772) facet_failover done wait for MDS to timeout and recover - unlinked 0 (time 1788973817 ; total 0 ; last 0) total: 400 unlinks in 7 seconds: 57.142857 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (137s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 13:10:32 (1788973832) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2068 1269236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:10:49 (1788973849) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:11:25 (1788973885) targets are mounted 13:11:25 (1788973885) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (83s) == replay-single test 44a: race in target handle connect ========================================================== 13:11:54 (1788973914) at_max=40 1 of 10 (1788973919) service : cur 5 worst 5 (at 1788970017, 3904s ago) 1 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 2 of 10 (1788973927) service : cur 5 worst 5 (at 1788970017, 3911s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 3 of 10 (1788973935) service : cur 6 worst 6 (at 1788973935, 1s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 4 of 10 (1788973942) service : cur 6 worst 6 (at 1788973935, 8s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 5 of 10 (1788973949) service : cur 6 worst 6 (at 1788973935, 16s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 6 of 10 (1788973957) service : cur 6 worst 6 (at 1788973935, 23s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 7 of 10 (1788973965) service : cur 6 worst 6 (at 1788973935, 31s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 8 of 10 (1788973972) service : cur 6 worst 6 (at 1788973935, 38s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 9 of 10 (1788973980) service : cur 6 worst 6 (at 1788973935, 46s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre 10 of 10 (1788973987) service : cur 6 worst 6 (at 1788973935, 53s ago) 6 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (89s) == replay-single test 44b: race in target handle connect ========================================================== 13:13:23 (1788974003) 1 of 10 (1788974005) service : cur 6 worst 6 (at 1788973935, 71s ago) 6 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 2 of 10 (1788974027) service : cur 6 worst 6 (at 1788973935, 94s ago) 6 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 3 of 10 (1788974050) service : cur 40 worst 40 (at 1788974047, 5s ago) 40 4 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 4 of 10 (1788974073) service : cur 40 worst 40 (at 1788974047, 28s ago) 1 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 5 of 10 (1788974096) service : cur 40 worst 40 (at 1788974047, 51s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 6 of 10 (1788974119) service : cur 40 worst 40 (at 1788974047, 74s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 7 of 10 (1788974141) service : cur 40 worst 40 (at 1788974047, 97s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 8 of 10 (1788974165) service : cur 40 worst 40 (at 1788974047, 120s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 9 of 10 (1788974187) service : cur 40 worst 40 (at 1788974047, 142s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre 10 of 10 (1788974210) service : cur 40 worst 40 (at 1788974047, 165s ago) 40 40 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.155@tcp:/lustre 7666232 3164 7210876 1% /mnt/lustre PASS 44b (236s) == replay-single test 44c: race in target handle connect ========================================================== 13:17:19 (1788974239) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1700 1269604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre total: 100 create in 1.50 seconds: 66.80 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:18:12 (1788974292) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:36 (1788974316) targets are mounted 13:18:36 (1788974316) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (95s) == replay-single test 45: Handle failed close ============ 13:18:54 (1788974334) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 /mnt/lustre/f45.replay-single has type file OK PASS 45 (9s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 13:19:03 (1788974343) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:19:28 (1788974368) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:19:54 (1788974394) targets are mounted 13:19:54 (1788974394) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (71s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 13:20:15 (1788974415) total: 20 open/close in 0.66 seconds: 30.21 ops/second Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 13:20:22 (1788974422) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 13:20:46 (1788974446) targets are mounted 13:20:46 (1788974446) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.43 seconds: 46.68 ops/second - unlinked 0 (time 1788974522 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (113s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 13:22:08 (1788974528) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2004 1269300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre total: 20 open/close in 0.50 seconds: 40.01 ops/second Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:22:23 (1788974543) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:23:00 (1788974580) targets are mounted 13:23:00 (1788974580) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.50 seconds: 39.68 ops/second - unlinked 0 (time 1788974643 ; total 0 ; last 0) total: 40 unlinks in 1 seconds: 40.000000 unlinks/second PASS 48 (122s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 13:24:11 (1788974651) PASS 50 (15s) == replay-single test 52: time out lock replay (3764) ==== 13:24:27 (1788974667) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.8237 fail_loc=0x80000157 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:24:34 (1788974674) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:24:59 (1788974699) targets are mounted 13:24:59 (1788974699) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (103s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 13:26:09 (1788974769) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:26:25 (1788974785) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:26:57 (1788974817) targets are mounted 13:26:57 (1788974817) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (63s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 13:27:12 (1788974832) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8237 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:27:27 (1788974847) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:28:01 (1788974881) targets are mounted 13:28:01 (1788974881) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (68s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 13:28:20 (1788974900) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:28:36 (1788974916) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:29:09 (1788974949) targets are mounted 13:29:09 (1788974949) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (58s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 13:29:18 (1788974958) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:29:27 (1788974967) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:30:00 (1788975000) targets are mounted 13:30:00 (1788975000) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (59s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 13:30:17 (1788975017) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:30:32 (1788975032) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:31:08 (1788975068) targets are mounted 13:31:08 (1788975068) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (67s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 13:31:24 (1788975084) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:31:41 (1788975101) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:32:17 (1788975137) targets are mounted 13:32:17 (1788975137) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (62s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 13:32:27 (1788975147) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:32:43 (1788975163) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:33:25 (1788975205) targets are mounted 13:33:25 (1788975205) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (68s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 13:33:35 (1788975215) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:33:53 (1788975233) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:34:25 (1788975265) targets are mounted 13:34:25 (1788975265) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (59s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 13:34:34 (1788975274) fail_loc=0x8000012b fail_loc=0x0 rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (69s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 13:35:43 (1788975343) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:35:56 (1788975356) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:36:28 (1788975388) targets are mounted 13:36:28 (1788975388) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (72s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 13:36:55 (1788975415) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2012 1269292 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:37:12 (1788975432) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:37:48 (1788975468) targets are mounted 13:37:48 (1788975468) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg355-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (80s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 13:38:16 (1788975496) fail_loc=0x8000012c - open/close 689 (time 1788975512.35 total 10.01 last 68.85) - open/close 1438 (time 1788975522.37 total 20.02 last 74.80) - open/close 2188 (time 1788975532.38 total 30.04 last 74.87) total: 2500 open/close in 34.70 seconds: 72.04 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:39:11 (1788975551) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:39:45 (1788975585) targets are mounted 13:39:45 (1788975585) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1788975622 ; total 0 ; last 0) total: 2500 unlinks in 21 seconds: 119.047623 unlinks/second PASS 58a (155s) == replay-single test 58b: test replay of setxattr op ==== 13:40:50 (1788975650) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1269012 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:41:05 (1788975665) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:41:42 (1788975702) targets are mounted 13:41:42 (1788975702) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) oleg355-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (80s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 13:42:10 (1788975730) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) PASS 58c (139s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 13:44:31 (1788975871) total: 200 open/close in 3.36 seconds: 59.61 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1269572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre - unlinked 0 (time 1788975891 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:44:58 (1788975898) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:45:29 (1788975929) targets are mounted 13:45:29 (1788975929) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788975939 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (74s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 13:45:45 (1788975945) total: 800 open/close in 9.26 seconds: 86.40 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1269084 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1936 1269368 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3164 7210876 1% /mnt/lustre - unlinked 0 (time 1788975970 ; total 0 ; last 0) total: 800 unlinks in 5 seconds: 160.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 13:46:21 (1788975981) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 13:46:50 (1788976010) targets are mounted 13:46:50 (1788976010) facet_failover done Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 13:47:04 (1788976024) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 13:47:28 (1788976048) targets are mounted 13:47:28 (1788976048) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (152s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 13:48:17 (1788976097) fail_loc=0x8000013a Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:48:23 (1788976103) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:48:47 (1788976127) targets are mounted 13:48:47 (1788976127) facet_failover done Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:49:01 (1788976141) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:49:33 (1788976173) targets are mounted 13:49:33 (1788976173) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143098 s, 286 kB/s PASS 61b (93s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 13:49:50 (1788976190) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 13:50:08 (1788976208) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 13:50:35 (1788976235) targets are mounted 13:50:35 (1788976235) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (62s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 13:50:52 (1788976252) Stopping /mnt/lustre-mds1 (opts:) on oleg355-server fail_loc=0x80000605 Start mgs: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg355-client: oleg355-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Start mgs: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (43s) == replay-single test 62: don't mis-drop resent replay === 13:51:35 (1788976295) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1269084 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1932 1269372 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3605432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3168 7210872 1% /mnt/lustre total: 25 open/close in 0.43 seconds: 57.56 ops/second fail_loc=0x80000707 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 13:51:50 (1788976310) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 13:52:22 (1788976342) targets are mounted 13:52:22 (1788976342) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1788976407 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (116s) == replay-single test 65a: AT: verify early replies ====== 13:53:31 (1788976411) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:3.0:1788976450.903091:0:129750:0:(client.c:582:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 36s (26s) req@ffff8ca10a4d2d00 x1875870924534656/t0(0) o101->lustre-MDT0000-mdc-ffff8ca1032a1000@192.168.203.155@tcp:12/10 lens 664/66320 e 1 to 0 dl 1788976486 ref 2 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 portal 12 : cur 11 worst 40 (at 1788974073, 2385s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1788970542, 5916s ago) 5 0 5 0 portal 23 : cur 5 worst 5 (at 1788970543, 5915s ago) 5 5 5 0 portal 30 : cur 5 worst 5 (at 1788970553, 5905s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1788970730, 5728s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1788971158, 5300s ago) 5 5 0 0 portal 13 : cur 5 worst 5 (at 1788971621, 4837s ago) 5 0 0 5 portal 12 : cur 5 worst 40 (at 1788974073, 2394s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1788970542, 5925s ago) 5 0 5 0 portal 23 : cur 5 worst 5 (at 1788970543, 5924s ago) 5 5 5 0 portal 30 : cur 5 worst 5 (at 1788970553, 5914s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1788970730, 5737s ago) 5 5 5 5 portal 24 : cur 5 worst 5 (at 1788971158, 5309s ago) 5 5 0 0 portal 13 : cur 5 worst 5 (at 1788971621, 4846s ago) 5 0 0 5 PASS 65a (61s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 13:54:32 (1788976472) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:1.0:1788976510.300497:0:2396:0:(client.c:582:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8ca106354f00 x1875870924556032/t0(0) o4->lustre-OST0000-osc-ffff8ca1032a1000@192.168.203.155@tcp:6/4 lens 4584/448 e 1 to 0 dl 1788976545 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 portal 28 : cur 5 worst 5 (at 1788970542, 5976s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1788970544, 5974s ago) 5 5 5 5 portal 17 : cur 5 worst 5 (at 1788970661, 5857s ago) 5 0 5 0 portal 6 : cur 36 worst 36 (at 1788976515, 3s ago) 36 0 0 0 PASS 65b (52s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 13:55:24 (1788976524) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1788974073, 2479s ago) 5 40 40 40 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1788974073, 2488s ago) 5 40 40 40 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 37 worst 40 (at 1788974073, 2501s ago) 37 40 40 40 fail_loc=0 portal 12 : cur 5 worst 40 (at 1788974073, 2512s ago) 37 40 40 40 Current MDT timeout 5, worst 40 PASS 66a (66s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:56:30 (1788976590) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (94s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:58:04 (1788976684) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (79s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:59:23 (1788976763) at_history=8 at_history=8 Creating to objid 5057 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.35 seconds: 51.12 ops/second Connected clients: oleg355-client.virtnet oleg355-client.virtnet service : cur 5 worst 5 (at 1788970053, 6743s ago) 1 1 1 0 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg355-client.virtnet oleg355-client.virtnet service : cur 5 worst 5 (at 1788970053, 6748s ago) 1 1 1 1 0 osc reconnect attempts on 2nd slow PASS 67b (42s) == replay-single test 68: AT: verify slowing locks ======= 14:00:05 (1788976805) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (83s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 14:01:28 (1788976888) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (3s) == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 14:01:31 (1788976891) Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg355-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=135409 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1268912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3597108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26160 3573860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27752 7170968 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:01:55 (1788976915) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 oleg355-client.virtnet: looking for dbench program oleg355-client.virtnet: /usr/bin/dbench oleg355-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg355-client.virtnet oleg355-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg355-client.virtnet' oleg355-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg355-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg355-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg355-client.virtnet at Wed Sep 9 14:01:35 EDT 2026 oleg355-client.virtnet: waiting for dbench pid 135440 oleg355-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg355-client.virtnet: oleg355-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg355-client.virtnet: failed to create barrier semaphore oleg355-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg355-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg355-client.virtnet: releasing clients oleg355-client.virtnet: 1 72 3.14 MB/sec warmup 1 sec latency 60.973 ms oleg355-client.virtnet: 1 147 3.14 MB/sec warmup 2 sec latency 136.153 ms oleg355-client.virtnet: 1 199 2.61 MB/sec warmup 3 sec latency 158.050 ms oleg355-client.virtnet: 1 276 2.57 MB/sec warmup 4 sec latency 105.667 ms oleg355-client.virtnet: 1 324 2.34 MB/sec warmup 5 sec latency 123.399 ms oleg355-client.virtnet: 1 388 2.30 MB/sec warmup 6 sec latency 98.952 ms oleg355-client.virtnet: 1 446 2.23 MB/sec warmup 7 sec latency 109.329 ms oleg355-client.virtnet: 1 511 2.21 MB/sec warmup 8 sec latency 104.342 ms oleg355-client.virtnet: 1 659 2.34 MB/sec warmup 9 sec latency 74.074 ms oleg355-client.virtnet: 1 664 2.11 MB/sec warmup 10 sec latency 956.565 ms oleg355-client.virtnet: 1 721 1.93 MB/sec warmup 11 sec latency 1224.900 ms oleg355-client.virtnet: 1 774 1.78 MB/sec warmup 12 sec latency 139.372 ms oleg355-client.virtnet: 1 811 1.65 MB/sec warmup 13 sec latency 96.306 ms oleg355-client.virtnet: 1 876 1.55 MB/sec warmup 14 sec latency 97.404 ms oleg355-client.virtnet: 1 892 1.44 MB/sec warmup 15 sec latency 397.708 ms oleg355-client.virtnet: 1 933 1.36 MB/sec warmup 16 sec latency 118.971 ms oleg355-client.virtnet: 1 986 1.29 MB/sec warmup 17 sec latency 178.677 ms oleg355-client.virtnet: 1 986 1.22 MB/sec warmup 18 sec latency 1179.373 ms oleg355-client.virtnet: 1 986 1.15 MB/sec warmup 19 sec latency 2179.561 ms oleg355-client.virtnet: 1 986 1.09 MB/sec warmup 20 sec latency 3179.778 ms oleg355-client.virtnet: 1 986 1.04 MB/sec warmup 21 sec latency 4181.652 ms oleg355-client.virtnet: 1 986 0.99 MB/sec warmup 22 sec latency 5183.098 ms oleg355-client.virtnet: 1 986 0.95 MB/sec warmup 23 sec latency 6183.430 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 1 sec latency 8184.707 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 2 sec latency 9184.906 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 3 sec latency 10185.068 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 4 sec latency 11186.530 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 5 sec latency 12187.842 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 6 sec latency 13189.440 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 7 sec latency 14190.975 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 8 sec latency 15191.216 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 9 sec latency 16191.671 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 10 seStart mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:02:28 (1788976948) targets are mounted 14:02:28 (1788976948) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13124 3593812 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36412 3569948 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49536 7163760 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server c latency 17191.967 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 11 sec latency 18192.148 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 12 sec latency 19194.995 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 13 sec latency 20195.205 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 14 sec latency 21195.433 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 15 sec latency 22199.944 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 16 sec latency 23200.411 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 17 sec latency 24201.400 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 18 sec latency 25201.652 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 19 sec latency 26203.680 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 20 sec latency 27203.937 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 21 sec latency 28204.829 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 22 sec latency 29205.119 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 23 sec latency 30205.358 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 24 sec latency 31208.102 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 25 sec latency 32208.421 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 26 sec latency 33208.778 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 27 sec latency 34209.153 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 28 sec latency 35213.188 ms oleg355-client.virtnet: 1 986 0.00 MB/sec execute 29 sec latency 36213.383 ms oleg355-client.virtnet: 1 1063 0.03 MB/sec execute 30 sec latency 36295.825 ms oleg355-client.virtnet: 1 1172 0.04 MB/sec execute 31 sec latency 61.925 ms oleg355-client.virtnet: 1 1240 0.04 MB/sec execute 32 sec latency 66.900 ms oleg355-client.virtnet: 1 1398 0.12 MB/sec execute 33 sec latency 81.038 ms oleg355-client.virtnet: 1 1486 0.12 MB/sec execute 34 sec latency 64.182 ms oleg355-client.virtnet: 1 1554 0.12 MB/sec execute 35 sec latency 71.083 ms oleg355-client.virtnet: 1 1643 0.12 MB/sec execute 36 sec latency 55.665 ms oleg355-client.virtnet: 1 1698 0.12 MB/sec execute 37 sec latency 71.750 ms oleg355-client.virtnet: 1 1820 0.12 MB/sec execute 38 sec latency 78.717 ms oleg355-client.virtnet: 1 1909 0.12 MB/sec execute 39 sec latency 95.941 ms oleg355-client.virtnet: 1 2010 0.12 MB/sec execute 40 sec latency 97.022 ms oleg355-client.virtnet: 1 2110 0.12 MB/sec execute 41 sec latency 62.093 ms oleg355-client.virtnet: 1 2198 0.12 MB/sec execute 42 sec latency 98.321 ms oleg355-client.virtnet: 1 2350 0.14 MB/sec execute 43 sec latency 94.268 ms oleg355-client.virtnet: 1 2412 0.14 MB/sec execute 44 sec latency 66.911 ms oleg355-client.virtnet: 1 2478 0.15 MB/sec execute 45 sec latency 92.969 ms oleg355-client.virtnet: 1 2594 0.15 MB/sec execute 46 sec latency 69.924 ms oleg355-client.virtnet: 1 2775 0.21 MB/sec execute 47 sec latency 91.175 ms oleg355-client.virtnet: 1 2983 0.24 MB/sec execute 48 sec latency 66.207 ms oleg355-client.virtnet: 1 3210 0.25 MB/sec execute 49 sec latency 56.031 ms oleg355-client.virtnet: 1 3375 0.27 MB/sec execute 50 sec latency 65.797 ms oleg355-client.virtnet: 1 3579 0.32 MB/sec execute 51 sec latency 71.828 ms oleg355-client.virtnet: 1 3678 0.32 MB/sec execute 52 sec latency 69.608 ms oleg355-client.virtnet: 1 3773 0.34 MB/sec execute 53 sec latency 79.092 ms oleg355-client.virtnet: 1 3833 0.33 MB/sec execute 54 sec latency 88.155 ms oleg355-14:02:55 (1788976975) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:03:23 (1788977003) targets are mounted 14:03:23 (1788977003) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec client.virtnet: 1 3927 0.33 MB/sec execute 55 sec latency 50.306 ms oleg355-client.virtnet: 1 4002 0.32 MB/sec execute 56 sec latency 93.015 ms oleg355-client.virtnet: 1 4069 0.32 MB/sec execute 57 sec latency 76.508 ms oleg355-client.virtnet: 1 4208 0.31 MB/sec execute 58 sec latency 61.416 ms oleg355-client.virtnet: 1 4288 0.31 MB/sec execute 59 sec latency 127.420 ms oleg355-client.virtnet: 1 4369 0.31 MB/sec execute 60 sec latency 90.857 ms oleg355-client.virtnet: 1 4458 0.31 MB/sec execute 61 sec latency 66.779 ms oleg355-client.virtnet: 1 4524 0.31 MB/sec execute 62 sec latency 81.186 ms oleg355-client.virtnet: 1 4610 0.32 MB/sec execute 63 sec latency 70.975 ms oleg355-client.virtnet: 1 4723 0.32 MB/sec execute 64 sec latency 71.003 ms oleg355-client.virtnet: 1 4808 0.31 MB/sec execute 65 sec latency 66.196 ms oleg355-client.virtnet: 1 4984 0.35 MB/sec execute 66 sec latency 67.340 ms oleg355-client.virtnet: 1 5066 0.35 MB/sec execute 67 sec latency 89.289 ms oleg355-client.virtnet: 1 5133 0.34 MB/sec execute 68 sec latency 89.717 ms oleg355-client.virtnet: 1 5184 0.34 MB/sec execute 69 sec latency 141.203 ms oleg355-client.virtnet: 1 5252 0.33 MB/sec execute 70 sec latency 81.740 ms oleg355-client.virtnet: 1 5296 0.33 MB/sec execute 71 sec latency 85.575 ms oleg355-client.virtnet: 1 5360 0.33 MB/sec execute 72 sec latency 86.322 ms oleg355-client.virtnet: 1 5438 0.32 MB/sec execute 73 sec latency 63.740 ms oleg355-client.virtnet: 1 5539 0.32 MB/sec execute 74 sec latency 68.955 ms oleg355-client.virtnet: 1 5579 0.32 MB/sec execute 75 sec latency 103.489 ms oleg355-client.virtnet: 1 5678 0.31 MB/sec execute 76 sec latency 92.181 ms oleg355-client.virtnet: 1 5748 0.31 MB/sec execute 77 sec latency 70.485 ms oleg355-client.virtnet: 1 5854 0.32 MB/sec execute 78 sec latency 91.664 ms oleg355-client.virtnet: 1 5941 0.32 MB/sec execute 79 sec latency 58.966 ms oleg355-client.virtnet: 1 6026 0.32 MB/sec execute 80 sec latency 63.536 ms oleg355-client.virtnet: 1 6144 0.32 MB/sec execute 81 sec latency 72.049 ms oleg355-client.virtnet: 1 6432 0.36 MB/sec execute 82 sec latency 59.653 ms oleg355-client.virtnet: 1 6539 0.37 MB/sec execute 83 sec latency 57.968 ms oleg355-client.virtnet: 1 6762 0.37 MB/sec execute 84 sec latency 79.336 ms oleg355-client.virtnet: 1 6964 0.39 MB/sec execute 85 sec latency 53.007 ms oleg355-client.virtnet: 1 7164 0.42 MB/sec execute 86 sec latency 57.419 ms oleg355-client.virtnet: 1 7246 0.41 MB/sec execute 87 sec latency 46.254 ms oleg355-client.virtnet: 1 7341 0.42 MB/sec execute 88 sec latency 63.068 ms oleg355-client.virtnet: 1 7412 0.41 MB/sec execute 89 sec latency 65.190 ms oleg355-client.virtnet: 1 7483 0.41 MB/sec execute 90 sec latency 73.016 ms oleg355-client.virtnet: 1 7558 0.41 MB/sec execute 91 sec latency 66.804 ms oleg355-client.virtnet: 1 7628 0.40 MB/sec execute 92 sec latency 71.322 ms oleg355-client.virtnet: 1 7730 0.40 MB/sec execute 93 sec latency 124.731 ms oleg355-client.virtnet: 1 7805 0.40 MB/sec execute 94 sec latency 95.637 ms oleg355-client.virtnet: 1 7883 0.40 MB/sec execute 95 sec latency 73.961 ms oleg355-client.virtnet: 1 7937 0.39 MB/sec execute 96 sec latency 120.159 ms oleg355-client.virtnet: 1 8031 0.39 MB/sec execute 97 sec latency 49.691 ms oleg355-client.virtnet: 1 8130 0.39 MB/sec execute 98 sec latency 65.532 ms oleg355-client.virtnet: 1 8263 0.40 MB/sec execute 99 sec latency 51.948 ms oleg355-client.virtnet: 1 8360 0.39 MB/sec execute 100 sec latency 74.927 ms oleg355-client.virtnet: 1 8535 0.42 MB/sec execute 101 sec latency 59.864 ms oleg355-client.virtnet: 1 8643 0.42 MB/sec execute 102 sec latency 67.350 ms oleg355-client.virtnet: 1 8730 0.41 MB/sec execute 103 sec latency 62.131 ms oleg355-client.virtnet: 1 8814 0.41 MB/sec execute 104 sec latency 94.689 ms oleg355-client.virtnet: 1 8893 0.41 MB/sec execute 105 sec latency 70.214 ms oleg355-client.virtnet: 1 8970 0.40 MB/sec execute 106 sec latency 63.258 ms oleg355-client.virtnet: 1 9097 0.40 MB/sec execute 107 sec latency 62.906 ms oleg355-client.virtnet: 1 9216 0.40 MB/sec execute 108 sec latency 58.765 ms oleg355-client.virtnet: 1 9355 0.39 MB/sec execute 109 sec latency 57.879 ms oleg355-client.virtnet: 1 9498 0.40 MB/sec execute 110 sec latency 50.103 ms oleg355-client.virtnet: 1 9593 0.41 MB/sec execute 111 sec latency 64.421 ms oleg355-client.virtnet: 1 9712 0.40 MB/sec execute 112 sec latency 54.817 ms oleg355-client.virtnet: 1 10036 0.44 MB/sec execute 113 sec latency 49.047 ms oleg355-client.virtnet: 1 10300 0.44 MB/sec execute 114 sec latency 58.762 ms oleg355-client.virtnet: 1 10530 0.46 MB/sec execute 115 sec latency 49.127 ms oleg355-client.virtnet: 1 10709 0.47 MB/sec execute 116 sec latency 49.873 ms oleg355-client.virtnet: 1 10810 0.47 MB/sec execute 117 sec latency 50.619 ms oleg355-client.virtnet: 1 10935 0.47 MB/sec execute 118 sec latency 48.399 ms oleg355-client.virtnet: 1 11025 0.47 MB/sec execute 119 sec latency 58.202 ms oleg355-client.virtnet: 1 cleanup 120 sec oleg355-client.virtnet: 1 cleanup 121 sec oleg355-client.virtnet: 0 cleanup 122 sec oleg355-client.virtnet: oleg355-client.virtnet: Operation Count AvgLat MaxLat oleg355-client.virtnet: ---------------------------------------- oleg355-client.virtnet: NTCreateX 1744 28.507 141.179 oleg355-client.virtnet: Close 1281 6.641 41.314 oleg355-client.virtnet: Rename 74 41.909 94.665 oleg355-client.virtnet: Unlink 333 10.071 38.733 oleg355-client.virtnet: Qpathinfo 1614 7.401 78.689 oleg355-client.virtnet: Qfileinfo 282 0.552 6.363 oleg355-client.virtnet: Qfsinfo 286 0.738 38.287 oleg355-client.virtnet: Sfileinfo 133 300.591 36295.801 oleg355-client.virtnet: Find 605 1.572 49.778 oleg355-client.virtnet: WriteX 861 5.305 55.197 oleg355-client.virtnet: ReadX 2799 0.276 51.239 oleg355-client.virtnet: LockX 6 5.152 8.647 oleg355-client.virtnet: UnlockX 6 4.594 9.767 oleg355-client.virtnet: Flush 105 35.625 127.404 oleg355-client.virtnet: oleg355-client.virtnet: Throughput 0.473206 MB/sec 1 clients 1 procs max_latency=36295.825 ms oleg355-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg355-client.virtnet at Wed Sep 9 14:04:01 EDT 2026 with return code 0 oleg355-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg355-client.virtnet oleg355-client.virtnet: /mnt/lustre/d70b.replay-single/oleg355-client.virtnet /mnt/lustre/d70b.replay-single/oleg355-client.virtnet oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg355-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg355-client.virtnet: removed directory 'clients/client0' oleg355-client.virtnet: removed directory 'clients' oleg355-client.virtnet: removed 'client.txt' oleg355-client.virtnet: /mnt/lustre/d70b.replay-single/oleg355-client.virtnet oleg355-client.virtnet: dbench successfully finished PASS 70b (157s) == replay-single test 70c: tar 2mdts recovery ============ 14:04:08 (1788977048) Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar 138332 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4824 1266480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4448 1266856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18840 3580912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2836 3600424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 21676 7181336 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:06:34 (1788977194) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:07:11 (1788977231) targets are mounted 14:07:11 (1788977231) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (247s) == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 14:08:15 (1788977295) Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started 140160 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4216 1267088 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3780 1267524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605408 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210836 1% /mnt/lustre test_70d fail mds2 1 times Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:10:43 (1788977443) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:11:10 (1788977470) targets are mounted 14:11:10 (1788977470) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 140160 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (191s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 14:11:27 (1788977487) debug=+ha Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started PID=144256 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3248 1268056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2896 1268408 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre test_70e fail mds2 1 times Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:13:53 (1788977633) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:14:22 (1788977662) targets are mounted 14:14:22 (1788977662) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (193s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 14:14:39 (1788977679) mount clients oleg355-client.virtnet ... Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3204 1268100 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2748 1268556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3596092 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3601316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7197408 1% /mnt/lustre ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 14:15:00 (1788977700) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Started lustre-OST0000 14:15:32 (1788977732) targets are mounted 14:15:32 (1788977732) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... ldlm.namespaces.MGC192.168.203.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8ca1032a1000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8ca1032a1000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg355-client.virtnet' ... PASS 70f (75s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 14:15:54 (1788977754) Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started 153982 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3484 1267820 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3240 1268064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3604 1267700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3364 1267940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre fail mds1 mds2 1 times Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:18:34 (1788977914) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 14:19:04 (1788977944) targets are mounted 14:19:04 (1788977944) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 153982 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (219s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 14:19:34 (1788977974) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1268288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2468 1268836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:19:49 (1788977989) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:20:21 (1788978021) targets are mounted 14:20:21 (1788978021) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (76s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 14:20:49 (1788978049) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8237 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1904 1269400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:21:09 (1788978069) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:21:38 (1788978098) targets are mounted 14:21:38 (1788978098) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (74s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 14:22:03 (1788978123) Stopping clients: oleg355-client.virtnet /mnt/lustre (opts:) Stopping client oleg355-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg355-server Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:22:15 (1788978135) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:22:38 (1788978158) targets are mounted 14:22:38 (1788978158) facet_failover done Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Mount client oleg355-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (60s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 14:23:03 (1788978183) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1268992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:23:18 (1788978198) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:23:51 (1788978231) targets are mounted 14:23:51 (1788978231) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.32 seconds: 62.25 ops/second PASS 80a (66s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 14:24:09 (1788978249) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:24:32 (1788978272) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:25:02 (1788978302) targets are mounted 14:25:02 (1788978302) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.42 seconds: 48.10 ops/second PASS 80b (72s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 14:25:21 (1788978321) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:25:41 (1788978341) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:26:14 (1788978374) targets are mounted 14:26:14 (1788978374) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:26:30 (1788978390) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:26:56 (1788978416) targets are mounted 14:26:56 (1788978416) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.33 seconds: 59.96 ops/second PASS 80c (112s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 14:27:13 (1788978433) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:27:53 (1788978473) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Failover mds2 to oleg355-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 14:28:38 (1788978518) targets are mounted 14:28:38 (1788978518) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.34 seconds: 58.33 ops/second PASS 80d (105s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 14:28:58 (1788978538) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:29:17 (1788978557) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:29:49 (1788978589) targets are mounted 14:29:49 (1788978589) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.28 seconds: 71.95 ops/second PASS 80e (68s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 14:30:06 (1788978606) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:30:20 (1788978620) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:30:50 (1788978650) targets are mounted 14:30:50 (1788978650) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.38 seconds: 53.17 ops/second PASS 80f (66s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 14:31:12 (1788978672) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:31:35 (1788978695) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:32:13 (1788978733) targets are mounted 14:32:13 (1788978733) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:32:31 (1788978751) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:33:01 (1788978781) targets are mounted 14:33:01 (1788978781) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.40 seconds: 49.51 ops/second PASS 80g (126s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 14:33:18 (1788978798) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:33:47 (1788978827) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 14:34:17 (1788978857) targets are mounted 14:34:17 (1788978857) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.37 seconds: 54.45 ops/second PASS 80h (85s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 14:34:43 (1788978883) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:35:07 (1788978907) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:35:32 (1788978932) targets are mounted 14:35:32 (1788978932) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (68s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 14:35:51 (1788978951) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:36:05 (1788978965) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:36:41 (1788979001) targets are mounted 14:36:41 (1788979001) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (67s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 14:36:58 (1788979018) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1960 1269344 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1960 1269344 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:37:21 (1788979041) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:37:58 (1788979078) targets are mounted 14:37:58 (1788979078) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:38:18 (1788979098) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:38:45 (1788979125) targets are mounted 14:38:45 (1788979125) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (125s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 14:39:04 (1788979144) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1964 1269340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:39:30 (1788979170) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 14:40:23 (1788979223) targets are mounted 14:40:23 (1788979223) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (113s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 14:40:56 (1788979256) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:41:19 (1788979279) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:41:53 (1788979313) targets are mounted 14:41:53 (1788979313) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (80s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 14:42:17 (1788979337) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1964 1269340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:42:32 (1788979352) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:43:00 (1788979380) targets are mounted 14:43:00 (1788979380) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (60s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 14:43:17 (1788979397) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:43:43 (1788979423) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:44:18 (1788979458) targets are mounted 14:44:18 (1788979458) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:44:33 (1788979473) shut down facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 14:45:01 (1788979501) targets are mounted 14:45:01 (1788979501) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (124s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 14:45:21 (1788979521) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3176 7210864 1% /mnt/lustre Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Failing mds2 on oleg355-server Stopping /mnt/lustre-mds2 (opts:) on oleg355-server 14:45:54 (1788979554) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server facet: mds2 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Failover mds2 to oleg355-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 14:46:25 (1788979585) targets are mounted 14:46:25 (1788979585) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (84s) == replay-single test 84a: stale open during export disconnect ========================================================== 14:46:45 (1788979605) fail_loc=0x80000144 total: 1 open/close in 0.10 seconds: 10.23 ops/second oleg355-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg355-client: oleg355-client: ssh exited with exit code 5 PASS 84a (19s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 14:47:04 (1788979624) before recovery: unused locks count = 204 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:47:24 (1788979644) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:47:58 (1788979678) targets are mounted 14:47:58 (1788979678) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 100 PASS 85a (74s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 14:48:18 (1788979698) before recovery: unused locks count = 100 Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 14:48:44 (1788979724) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 14:49:12 (1788979752) targets are mounted 14:49:12 (1788979752) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (76s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 14:49:34 (1788979774) Stopping clients: oleg355-client.virtnet /mnt/lustre (opts:) Stopping client oleg355-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Mount client oleg355-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre Started clients oleg355-client.virtnet: 192.168.203.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) PASS 86 (32s) == replay-single test 87a: write replay ================== 14:50:06 (1788979806) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1792 3605228 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1784 3605236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3576 7210464 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.574762 s, 14.6 MB/s Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 14:50:24 (1788979824) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 14:50:56 (1788979856) targets are mounted 14:50:57 (1788979857) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.514158 s, 16.3 MB/s PASS 87a (73s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 14:51:19 (1788979879) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9984 3597036 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1784 3605236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11768 7202272 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.620309 s, 13.5 MB/s 8+0 records in 8+0 records out 8 bytes copied, 0.0111619 s, 0.7 kB/s Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 14:51:41 (1788979901) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 14:52:14 (1788979934) targets are mounted 14:52:14 (1788979934) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes copied, 0.0240682 s, 3.0 kB/s PASS 87b (74s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 14:52:34 (1788979954) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9988 3597032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1784 3605236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11772 7202268 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1268884 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9988 3597032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1784 3605236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11772 7202268 1% /mnt/lustre before test: last_id = 7041, next_id = 7012 Creating to objid 7041 on ost lustre-OST0000... total: 31 open/close in 0.72 seconds: 42.96 ops/second total: 8 open/close in 0.14 seconds: 56.75 ops/second before recovery: last_id = 7073, next_id = 7050 Stopping /mnt/lustre-mds1 (opts:) on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server Failover mds1 to oleg355-server oleg355-server.virtnet Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg355-server oleg355-server.virtnet Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 7081, next_id = 7050 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.321057 s, 1.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0916229 s, 5.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.166735 s, 3.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.207959 s, 2.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.11714 s, 4.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.167987 s, 3.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0833083 s, 6.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.161075 s, 3.3 MB/s -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7012 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7013 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7014 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7015 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7016 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7017 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7018 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7019 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7020 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7021 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7022 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7023 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7024 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7025 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7026 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7027 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7028 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7029 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7030 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7031 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7032 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7033 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7034 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7035 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7036 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7037 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7038 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7039 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7040 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7041 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7042 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7043 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7044 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7045 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7046 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7047 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7048 -rw-r--r-- 1 root root 0 Sep 9 14:52 /mnt/lustre/d88.replay-single/f-7049 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7053 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7054 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7055 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7056 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7057 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7058 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7059 -rw-r--r-- 1 root root 524288 Sep 9 14:54 /mnt/lustre/d88.replay-single/f-7060 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.218606 s, 2.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.144404 s, 3.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.163216 s, 3.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0808928 s, 6.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.157175 s, 3.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0972313 s, 5.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.196553 s, 2.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.159038 s, 3.3 MB/s PASS 88 (135s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 14:54:48 (1788980088) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. Waiting for MDT destroys to complete 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.51614 s, 20.3 MB/s -rw-r--r-- 1 root root 10485760 Sep 9 14:55 /mnt/lustre/d89.replay-single/f89.replay-single Stopping /mnt/lustre-ost1 (opts:) on oleg355-server Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:55:10 (1788980110) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 14:55:47 (1788980147) targets are mounted 14:55:47 (1788980147) facet_failover done Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 54 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7646268 free_after: 7646268 PASS 89 (160s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 14:57:28 (1788980248) Create the files Fail ost1 lustre-OST0000_UUID, display the list of affected files Stopping /mnt/lustre-ost1 (opts:) on oleg355-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/f0 /mnt/lustre/d90.replay-single/all Querying files on shutdown ost1: lfs find --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/f0 /mnt/lustre/d90.replay-single/all Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0000_UUID /mnt/lustre/d90.replay-single/f0 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 7083 0x1bab 0x280000401 * /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 0 7082 0x1baa 0x280000401 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Failover ost1 to oleg355-server oleg355-server.virtnet Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 90 (45s) == replay-single test 93a: replay + reconnect ============ 14:58:13 (1788980293) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0157532 s, 65.0 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg355-server Stopping /mnt/lustre-ost1 (opts:) on oleg355-server 14:58:21 (1788980301) shut down facet: ost1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover ost1 to oleg355-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 14:58:48 (1788980328) targets are mounted 14:58:48 (1788980328) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (86s) == replay-single test 93b: replay + reconnect on mds ===== 14:59:39 (1788980379) total: 20 open/close in 0.61 seconds: 32.87 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 14:59:49 (1788980389) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 15:00:23 (1788980423) targets are mounted 15:00:23 (1788980423) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (139s) == replay-single test complete, duration 10392 sec ======= 15:01:59 (1788980519) === replay-single: start cleanup 15:02:02 (1788980522) === === replay-single: finish cleanup 15:02:16 (1788980536) ===