-----============= acceptance-small: replay-single ============----- Tue Apr 14 11:27:23 EDT 2026 oleg254-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 oleg254-client.virtnet: executing check_config_client /mnt/lustre oleg254-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg254-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b1350063800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b1350063800.idle_timeout=debug disable quota as required oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 == replay-single test 0a: empty replay =================== 11:28:03 (1776180483) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (65s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:29:09 (1776180549) Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.50 seconds: 39.64 ops/second - unlinked 0 (time 1776180586 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (44s) == replay-single test 0c: check replay-barrier =========== 11:29:52 (1776180592) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (104s) == replay-single test 0d: expired recovery with no clients ========================================================== 11:31:37 (1776180697) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre PASS 0d (108s) == replay-single test 1: simple create =================== 11:33:25 (1776180805) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (63s) == replay-single test 2a: touch ========================== 11:34:28 (1776180868) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (70s) == replay-single test 2b: touch ========================== 11:35:38 (1776180938) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (65s) == replay-single test 2c: setstripe replay =============== 11:36:43 (1776181003) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (74s) == replay-single test 2d: setdirstripe replay ============ 11:37:57 (1776181077) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (63s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:39:00 (1776181140) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (71s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:40:11 (1776181211) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (53s) == replay-single test 3b: replay failed open -ENOMEM ===== 11:41:04 (1776181264) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (50s) == replay-single test 3c: replay failed open -ENOMEM ===== 11:41:54 (1776181314) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (52s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:42:46 (1776181366) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (64s) == replay-single test 4b: |x| rm 10 files ================ 11:43:50 (1776181430) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (57s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:44:47 (1776181487) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7518208 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (75s) == replay-single test 6a: mkdir + contained create ======= 11:46:03 (1776181563) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (61s) == replay-single test 6b: |X| rmdir ====================== 11:47:04 (1776181624) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (57s) == replay-single test 7: mkdir |X| contained create ====== 11:48:01 (1776181681) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (57s) == replay-single test 8: creat open |X| close ============ 11:48:58 (1776181738) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (66s) == replay-single test 9: |X| create (same inum/gen) ====== 11:50:04 (1776181804) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (62s) == replay-single test 10: create |X| rename unlink ======= 11:51:07 (1776181867) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (64s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:52:11 (1776181931) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre new old Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (61s) == replay-single test 12: open, unlink |X| close ========= 11:53:12 (1776181992) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (53s) == replay-single test 13: open chmod 0 |x| write close === 11:54:05 (1776182045) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (56s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:55:01 (1776182101) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (58s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:55:59 (1776182159) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (46s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:56:46 (1776182206) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (53s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:57:40 (1776182260) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (56s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:58:37 (1776182317) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 pid: 50702 will close Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (56s) == replay-single test 19: mcreate, open, write, rename === 11:59:32 (1776182372) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre old Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (59s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:00:31 (1776182431) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (55s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 12:01:26 (1776182486) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1058 0x422 0 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 4356+0 records in 4355+0 records out 17838080 bytes (18 MB, 17 MiB) copied, 3.19894 s, 5.6 MB/s Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg254-server: *.lustre-MDT0000.recovery_status status: COMPLETE sleep 5 for ZFS MDS Waiting for MDT destroys to complete before 6144, after 6144 PASS 20b (86s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 12:02:52 (1776182572) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 -rw-r--r-- 1 root root 1 Apr 14 12:02 /mnt/lustre/f20c.replay-single PASS 20c (12s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:03:04 (1776182584) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (60s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:04:04 (1776182644) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (57s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:05:01 (1776182701) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (62s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 12:06:03 (1776182763) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (67s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:07:10 (1776182830) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (62s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:08:12 (1776182892) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (64s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:09:16 (1776182956) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (60s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:10:16 (1776183016) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (60s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:11:16 (1776183076) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (60s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:12:16 (1776183136) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (59s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:13:15 (1776183195) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (62s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:14:18 (1776183258) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 PASS 32 (11s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:14:29 (1776183269) total: 10 open/close in 0.34 seconds: 29.43 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.22 seconds: 45.30 ops/second PASS 33a (30s) == replay-single test 33b: test fid seq allocation ======= 12:14:59 (1776183299) fail_loc=0x1311 total: 10 open/close in 0.25 seconds: 40.55 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.19 seconds: 52.71 ops/second PASS 33b (28s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 12:15:27 (1776183327) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (27s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 12:15:55 (1776183355) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (27s) == replay-single test 36: don't resend cancel ============ 12:16:23 (1776183383) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 36 (47s) == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 12:17:09 (1776183429) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (33s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:17:42 (1776183462) - open/close 750 (time 1776183475.81 total 10.01 last 74.91) total: 800 open/close in 10.83 seconds: 73.89 ops/second - unlinked 0 (time 1776183480 ; total 0 ; last 0) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776183537 ; total 0 ; last 0) total: 400 unlinks in 5 seconds: 80.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (88s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:19:11 (1776183551) - open/close 752 (time 1776183564.66 total 10.01 last 75.15) total: 800 open/close in 10.79 seconds: 74.15 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776183572 ; total 0 ; last 0) total: 400 unlinks in 4 seconds: 100.000000 unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776183620 ; total 0 ; last 0) total: 400 unlinks in 7 seconds: 57.142857 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (85s) == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 12:20:36 (1776183636) SKIP: replay-single test_40 layout_lock needs MDS connection for IO SKIP 40 (3s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:20:38 (1776183638) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00532082 s, 770 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281113 s, 146 kB/s PASS 41 (9s) == replay-single test 42: recovery after ost failure ===== 12:20:47 (1776183647) - open/close 779 (time 1776183659.93 total 10.00 last 77.90) total: 800 open/close in 10.29 seconds: 77.76 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776183666 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second debug=-1 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 wait for MDS to timeout and recover - unlinked 0 (time 1776183743 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (105s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:22:32 (1776183752) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (61s) == replay-single test 44a: race in target handle connect ========================================================== 12:23:33 (1776183813) at_max=40 1 of 10 (1776183817) service : cur 29 worst 29 (at 1776183779, 38s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 2 of 10 (1776183824) service : cur 29 worst 29 (at 1776183779, 45s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 3 of 10 (1776183831) service : cur 29 worst 29 (at 1776183779, 51s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 4 of 10 (1776183837) service : cur 29 worst 29 (at 1776183779, 58s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 5 of 10 (1776183843) service : cur 29 worst 29 (at 1776183779, 65s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 6 of 10 (1776183851) service : cur 29 worst 29 (at 1776183779, 72s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 7 of 10 (1776183858) service : cur 29 worst 29 (at 1776183779, 79s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 8 of 10 (1776183864) service : cur 29 worst 29 (at 1776183779, 86s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 9 of 10 (1776183872) service : cur 29 worst 29 (at 1776183779, 93s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 10 of 10 (1776183879) service : cur 29 worst 29 (at 1776183779, 101s ago) 29 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (79s) == replay-single test 44b: race in target handle connect ========================================================== 12:24:52 (1776183892) 1 of 10 (1776183893) service : cur 29 worst 29 (at 1776183779, 115s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 2 of 10 (1776183902) service : cur 29 worst 29 (at 1776183779, 123s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 3 of 10 (1776183903) service : cur 29 worst 29 (at 1776183779, 124s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 4 of 10 (1776183905) service : cur 29 worst 29 (at 1776183779, 126s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 5 of 10 (1776183907) service : cur 29 worst 29 (at 1776183779, 128s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 6 of 10 (1776183909) service : cur 29 worst 29 (at 1776183779, 130s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 7 of 10 (1776183911) service : cur 29 worst 29 (at 1776183779, 132s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 8 of 10 (1776183912) service : cur 29 worst 29 (at 1776183779, 133s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 9 of 10 (1776183914) service : cur 29 worst 29 (at 1776183779, 135s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress 10 of 10 (1776183916) service : cur 29 worst 29 (at 1776183779, 137s ago) 29 0 0 0 fail_loc=0x80000704 df: /mnt/lustre: Operation already in progress PASS 44b (29s) == replay-single test 44c: race in target handle connect ========================================================== 12:25:21 (1776183921) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 100 create in 0.99 seconds: 101.44 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (77s) == replay-single test 45: Handle failed close ============ 12:26:38 (1776183998) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 /mnt/lustre/f45.replay-single has type file OK PASS 45 (7s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:26:45 (1776184005) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (61s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:27:46 (1776184066) total: 20 open/close in 0.42 seconds: 47.32 ops/second Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.31 seconds: 63.82 ops/second - unlinked 0 (time 1776184170 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (110s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:29:36 (1776184176) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 20 open/close in 0.36 seconds: 56.10 ops/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x80000216 total: 20 open/close in 0.42 seconds: 47.70 ops/second - unlinked 0 (time 1776184281 ; total 0 ; last 0) total: 40 unlinks in 1 seconds: 40.000000 unlinks/second PASS 48 (112s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:31:28 (1776184288) PASS 50 (15s) == replay-single test 52: time out lock replay (3764) ==== 12:31:43 (1776184303) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7128 fail_loc=0x80000157 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (64s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:32:47 (1776184367) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (71s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:33:58 (1776184438) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7128 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (70s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:35:08 (1776184508) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (52s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:36:00 (1776184560) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (62s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:37:02 (1776184622) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (74s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:38:16 (1776184696) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (62s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:39:18 (1776184758) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (56s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:40:14 (1776184814) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (61s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:41:15 (1776184875) fail_loc=0x8000012b touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory fail_loc=0x0 rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (16s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:41:31 (1776184891) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (76s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:42:47 (1776184967) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg254-server: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg254-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (81s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:44:08 (1776185048) fail_loc=0x8000012c - open/close 641 (time 1776185063.23 total 10.01 last 64.02) - open/close 1284 (time 1776185073.23 total 20.01 last 64.30) - open/close 2051 (time 1776185083.24 total 30.02 last 76.68) total: 2500 open/close in 35.17 seconds: 71.09 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1776185186 ; total 0 ; last 0) total: 2500 unlinks in 23 seconds: 108.695656 unlinks/second PASS 58a (167s) == replay-single test 58b: test replay of setxattr op ==== 12:46:55 (1776185215) Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) oleg254-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (67s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:48:02 (1776185282) Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) PASS 58c (106s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:49:49 (1776185389) total: 200 open/close in 2.97 seconds: 67.30 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776185401 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776185448 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (65s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:50:54 (1776185454) - open/close 785 (time 1776185469.05 total 10.01 last 78.43) total: 800 open/close in 10.17 seconds: 78.69 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1776185478 ; total 0 ; last 0) total: 800 unlinks in 5 seconds: 160.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (142s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:53:16 (1776185596) fail_loc=0x8000013a Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109836 s, 373 kB/s PASS 61b (107s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:55:03 (1776185703) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (59s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:56:02 (1776185762) Stopping /mnt/lustre-mds1 (opts:) on oleg254-server fail_loc=0x80000605 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 95 Start of lustre-mdt1/mdt1 on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (30s) == replay-single test 62: don't mis-drop resent replay === 12:56:32 (1776185792) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.43 seconds: 58.69 ops/second fail_loc=0x80000707 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1776185887 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (99s) == replay-single test 65a: AT: verify early replies ====== 12:58:11 (1776185891) at_history=8 at_history=8 debug=other fail_val=6000 fail_loc=0x8000050a 00000100:00001000:2.0:1776185923.178325:0:125864:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@0000000025d6ee50 x1862460027390592/t0(0) o101->lustre-MDT0000-mdc-ffff8b1350791000@192.168.202.154@tcp:12/10 lens 664/66264 e 1 to 0 dl 1776185954 ref 2 fl Rpc:PQr/0/ffffffff rc 0/-1 job:'createmany.0' portal 12 : cur 6 worst 31 (at 1776181014, 4915s ago) 31 28 28 28 portal 23 : cur 1 worst 10 (at 1776180918, 5011s ago) 1 10 1 1 portal 30 : cur 1 worst 1 (at 1776180807, 5122s ago) 1 0 0 0 portal 17 : cur 1 worst 1 (at 1776181008, 4921s ago) 1 1 1 1 portal 12 : cur 1 worst 31 (at 1776181014, 4925s ago) 31 28 28 28 portal 23 : cur 1 worst 10 (at 1776180918, 5021s ago) 1 10 1 1 portal 30 : cur 1 worst 1 (at 1776180807, 5132s ago) 1 0 0 0 portal 17 : cur 1 worst 1 (at 1776181008, 4931s ago) 1 1 1 1 PASS 65a (52s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:59:03 (1776185943) at_history=8 at_history=8 debug=other trace fail_val=6 fail_loc=0x224 fail_loc=0 00000100:00001000:3.0:1776185974.760554:0:2219:0:(client.c:514:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 31s (25s) req@00000000e4c98c04 x1862460027398656/t0(0) o4->lustre-OST0000-osc-ffff8b1350791000@192.168.202.154@tcp:6/4 lens 4584/448 e 1 to 0 dl 1776186005 ref 2 fl Rpc:Qr/0/ffffffff rc 0/-1 job:'multiop.0' portal 28 : cur 1 worst 31 (at 1776185487, 495s ago) 1 31 31 31 portal 7 : cur 1 worst 1 (at 1776180799, 5183s ago) 1 1 1 1 portal 17 : cur 1 worst 1 (at 1776180934, 5048s ago) 1 0 1 0 portal 6 : cur 31 worst 31 (at 1776181546, 4436s ago) 31 0 0 0 PASS 65b (43s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:59:46 (1776185986) at_history=8 at_history=8 portal 12 : cur 1 worst 31 (at 1776181014, 4999s ago) 31 28 28 28 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 31 worst 31 (at 1776181014, 5006s ago) 31 28 28 28 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 10 worst 31 (at 1776181014, 5019s ago) 31 28 28 28 fail_loc=0 portal 12 : cur 1 worst 31 (at 1776181014, 5030s ago) 31 28 28 28 Current MDT timeout 1, worst 31 PASS 66a (67s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:00:53 (1776186053) at_history=8 at_history=8 fail_val=6 fail_loc=0x50c fail_loc=0 network timeout orig 1, cur 7, worst 7 PASS 66b (57s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:01:50 (1776186110) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (64s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:02:54 (1776186174) at_history=8 at_history=8 Creating to objid 5537 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 17 open/close in 0.27 seconds: 61.87 ops/second Connected clients: oleg254-client.virtnet oleg254-client.virtnet service : cur 1 worst 1 (at 1776180380, 5828s ago) 1 0 0 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg254-client.virtnet oleg254-client.virtnet service : cur 1 worst 1 (at 1776180380, 5833s ago) 1 0 0 1 0 osc reconnect attempts on 2nd slow PASS 67b (44s) == replay-single test 68: AT: verify slowing locks ======= 13:03:38 (1776186218) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (84s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 13:05:02 (1776186302) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (3s) == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 13:05:05 (1776186305) Starting client oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg254-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=131389 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3712 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3721216 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7457792 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-client.virtnet: looking for dbench program oleg254-client.virtnet: /usr/bin/dbench oleg254-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg254-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg254-client.virtnet' oleg254-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg254-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg254-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Apr 14 13:05:09 EDT 2026 oleg254-client.virtnet: waiting for dbench pid 131420 oleg254-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg254-client.virtnet: oleg254-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg254-client.virtnet: failed to create barrier semaphore oleg254-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg254-client.virtnet: releasing clients oleg254-client.virtnet: 1 121 5.48 MB/sec warmup 1 sec latency 115.946 ms oleg254-client.virtnet: 1 181 3.65 MB/sec warmup 2 sec latency 140.797 ms oleg254-client.virtnet: 1 304 3.73 MB/sec warmup 3 sec latency 65.183 ms oleg254-client.virtnet: 1 380 3.39 MB/sec warmup 4 sec latency 125.062 ms oleg254-client.virtnet: 1 451 3.16 MB/sec warmup 5 sec latency 112.119 ms oleg254-client.virtnet: 1 522 3.00 MB/sec warmup 6 sec latency 112.306 ms oleg254-client.virtnet: 1 665 3.02 MB/sec warmup 7 sec latency 112.373 ms oleg254-client.virtnet: 1 718 2.66 MB/sec warmup 8 sec latency 93.622 ms oleg254-client.virtnet: 1 759 2.37 MB/sec warmup 9 sec latency 184.934 ms oleg254-client.virtnet: 1 791 2.14 MB/sec warmup 10 sec latency 304.511 ms oleg254-client.virtnet: 1 857 1.96 MB/sec warmup 11 sec latency 139.997 ms oleg254-client.virtnet: 1 910 1.81 MB/sec warmup 12 sec latency 219.878 ms oleg254-client.virtnet: 1 917 1.67 MB/sec warmup 13 sec latency 720.455 ms oleg254-client.virtnet: 1 917 1.55 MB/sec warmup 14 sec latency 1726.519 ms oleg254-client.virtnet: 1 917 1.45 MB/sec warmup 15 sec latency 2728.827 ms oleg254-client.virtnet: 1 917 1.36 MB/sec warmup 16 sec latency 3729.733 ms oleg254-client.virtnet: 1 917 1.28 MB/sec warmup 17 sec latency 4730.184 ms oleg254-client.virtnet: 1 917 1.21 MB/sec warmup 18 sec latency 5730.901 ms oleg254-client.virtnet: 1 917 1.14 MB/sec warmup 19 sec latency 6731.189 ms oleg254-client.virtnet: 1 917 1.09 MB/sec warmup 20 sec latency 7735.461 ms oleg254-client.virtnet: 1 917 1.03 MB/sec warmup 21 sec latency 8736.225 ms oleg254-client.virtnet: 1 917 0.99 MB/sec warmup 22 sec latency 9736.985 ms oleg254-client.virtnet: 1 917 0.94 MB/sec warmup 23 sec latency 10737.287 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 1 sec latency 12743.649 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 2 sec latency 13749.195 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 3 sec latency 14750.116 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 4 sec latency 15750.878 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 5 sec latency 16751.132 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 6 sec latency 17751.696 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 7 sec latency 18751.885 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 8 sec latency 19752.140 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 9 sec latency 20754.315 ms oleg254-client.virtnet: 1 917 0.00 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7460864 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server MB/sec execute 10 sec latency 21754.528 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 11 sec latency 22754.732 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 12 sec latency 23754.933 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 13 sec latency 24755.138 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 14 sec latency 25756.307 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 15 sec latency 26756.530 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 16 sec latency 27756.736 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 17 sec latency 28756.966 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 18 sec latency 29757.968 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 19 sec latency 30758.793 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 20 sec latency 31759.088 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 21 sec latency 32759.997 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 22 sec latency 33761.037 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 23 sec latency 34761.287 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 24 sec latency 35761.455 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 25 sec latency 36762.832 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 26 sec latency 37763.031 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 27 sec latency 38763.931 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 28 sec latency 39764.366 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 29 sec latency 40766.594 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 30 sec latency 41767.883 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 31 sec latency 42768.151 ms oleg254-client.virtnet: 1 917 0.00 MB/sec execute 32 sec latency 43768.918 ms oleg254-client.virtnet: 1 960 0.00 MB/sec execute 33 sec latency 43898.053 ms oleg254-client.virtnet: 1 995 0.00 MB/sec execute 34 sec latency 255.145 ms oleg254-client.virtnet: 1 1117 0.03 MB/sec execute 35 sec latency 76.448 ms oleg254-client.virtnet: 1 1181 0.03 MB/sec execute 36 sec latency 88.614 ms oleg254-client.virtnet: 1 1262 0.04 MB/sec execute 37 sec latency 72.969 ms oleg254-client.virtnet: 1 1426 0.11 MB/sec execute 38 sec latency 84.479 ms oleg254-client.virtnet: 1 1496 0.11 MB/sec execute 39 sec latency 229.961 ms oleg254-client.virtnet: 1 1574 0.11 MB/sec execute 40 sec latency 59.550 ms oleg254-client.virtnet: 1 1679 0.11 MB/sec execute 41 sec latency 134.381 ms oleg254-client.virtnet: 1 1779 0.11 MB/sec execute 42 sec latency 84.995 ms oleg254-client.virtnet: 1 1884 0.11 MB/sec execute 43 sec latency 134.091 ms oleg254-client.virtnet: 1 1982 0.11 MB/sec execute 44 sec latency 188.862 ms oleg254-client.virtnet: 1 2098 0.11 MB/sec execute 45 sec latency 71.858 ms oleg254-client.virtnet: 1 2191 0.11 MB/sec execute 46 sec latency 65.789 ms oleg254-client.virtnet: 1 2345 0.13 MB/sec execute 47 sec latency 74.299 ms oleg254-client.virtnet: 1 2378 0.13 MB/sec execute 48 sec latency 244.995 ms oleg254-client.virtnet: 1 2416 0.13 MB/sec execute 49 sec latency 162.357 ms oleg254-client.virtnet: 1 2468 0.14 MB/sec execute 50 sec latency 214.104 ms oleg254-client.virtnet: 1 2577 0.14 MB/sec execute 51 sec latency 171.618 ms oleg254-client.virtnet: 1 2762 0.20 MB/sec execute 52 sec latency 208.978 ms oleg254-client.virtnet: 1 2762 0.19 MB/sec execute 53 sec latency 1209.213 ms oleg254-client.virtnet: 1 2762 0.19 MB/sec reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 54 sec latency 2209.802 ms oleg254-client.virtnet: 1 2762 0.18 MB/sec execute 55 sec latency 3211.163 ms oleg254-client.virtnet: 1 2762 0.18 MB/sec execute 56 sec latency 4212.418 ms oleg254-client.virtnet: 1 2762 0.18 MB/sec execute 57 sec latency 5214.620 ms oleg254-client.virtnet: 1 2762 0.18 MB/sec execute 58 sec latency 6214.836 ms oleg254-client.virtnet: 1 2762 0.17 MB/sec execute 59 sec latency 7215.104 ms oleg254-client.virtnet: 1 2762 0.17 MB/sec execute 60 sec latency 8215.375 ms oleg254-client.virtnet: 1 2762 0.17 MB/sec execute 61 sec latency 9215.604 ms oleg254-client.virtnet: 1 2762 0.16 MB/sec execute 62 sec latency 10215.942 ms oleg254-client.virtnet: 1 2762 0.16 MB/sec execute 63 sec latency 11216.889 ms oleg254-client.virtnet: 1 2762 0.16 MB/sec execute 64 sec latency 12217.080 ms oleg254-client.virtnet: 1 2762 0.16 MB/sec execute 65 sec latency 13218.550 ms oleg254-client.virtnet: 1 2762 0.15 MB/sec execute 66 sec latency 14219.629 ms oleg254-client.virtnet: 1 2762 0.15 MB/sec execute 67 sec latency 15220.454 ms oleg254-client.virtnet: 1 2762 0.15 MB/sec execute 68 sec latency 16220.694 ms oleg254-client.virtnet: 1 2762 0.15 MB/sec execute 69 sec latency 17222.352 ms oleg254-client.virtnet: 1 2762 0.15 MB/sec execute 70 sec latency 18222.566 ms oleg254-client.virtnet: 1 2762 0.14 MB/sec execute 71 sec latency 19222.896 ms oleg254-client.virtnet: 1 2762 0.14 MB/sec execute 72 sec latency 20223.331 ms oleg254-client.virtnet: 1 2762 0.14 MB/sec execute 73 sec latency 21224.520 ms oleg254-client.virtnet: 1 2762 0.14 MB/sec execute 74 sec latency 22226.009 ms oleg254-client.virtnet: 1 2762 0.14 MB/sec execute 75 sec latency 23226.233 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 76 sec latency 24226.908 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 77 sec latency 25230.201 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 78 sec latency 26230.507 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 79 sec latency 27231.826 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 80 sec latency 28232.493 ms oleg254-client.virtnet: 1 2762 0.13 MB/sec execute 81 sec latency 29234.767 ms oleg254-client.virtnet: 1 2762 0.12 MB/sec execute 82 sec latency 30234.922 ms oleg254-client.virtnet: 1 2762 0.12 MB/sec execute 83 sec latency 31236.002 ms oleg254-client.virtnet: 1 2762 0.12 MB/sec execute 84 sec latency 32236.229 ms oleg254-client.virtnet: 1 2762 0.12 MB/sec execute 85 sec latency 33236.555 ms oleg254-client.virtnet: 1 2762 0.12 MB/sec execute 86 sec latency 34236.809 ms oleg254-client.virtnet: 1 2874 0.12 MB/sec execute 87 sec latency 34433.009 ms oleg254-client.virtnet: 1 2932 0.13 MB/sec execute 88 sec latency 250.404 ms oleg254-client.virtnet: 1 3012 0.13 MB/sec execute 89 sec latency 173.862 ms oleg254-client.virtnet: 1 3210 0.14 MB/sec execute 90 sec latency 265.307 ms oleg254-client.virtnet: 1 3339 0.15 MB/sec execute 91 sec latency 121.247 ms oleg254-client.virtnet: 1 3410 0.15 MB/sec execute 92 sec latency 96.920 ms oleg254-client.virtnet: 1 3576 0.18 MB/sec execute 93 sec latency 59.385 ms oleg254-client.virtnet: 1 3685 0.18 MB/sec execute 94 sec latency 63.671 ms oleg254-client.virtnet: 1 3786 0.19 MB/sec execute 95 sec latency 71.383 ms oleg254-client.virtnet: 1 3860 0.19 MB/sec execute 96 sec latency 134.452 ms oleg254-client.virtnet: 1 3932 0.19 MB/sec execute 97 sec latency 80.285 ms oleg254-client.virtnet: 1 3995 0.19 MB/sec execute 98 sec latency 92.793 ms oleg254-client.virtnet: 1 4159 0.18 MB/sec execute 99 sec latency 58.592 ms oleg254-client.virtnet: 1 4250 0.19 MB/sec execute 100 sec latency 236.387 ms oleg254-client.virtnet: 1 4349 0.19 MB/sec execute 101 sec latency 99.693 ms oleg254-client.virtnet: 1 4446 0.19 MB/sec execute 102 sec latency 111.278 ms oleg254-client.virtnet: 1 4505 0.19 MB/sec execute 103 sec latency 165.216 ms oleg254-client.virtnet: 1 4628 0.19 MB/sec execute 104 sec latency 115.588 ms oleg254-client.virtnet: 1 4714 0.19 MB/sec execute 105 sec latency 54.680 ms oleg254-client.virtnet: 1 4818 0.19 MB/sec execute 106 sec latency 85.647 ms oleg254-client.virtnet: 1 5012 0.22 MB/sec execute 107 sec latency 89.306 ms oleg254-client.virtnet: 1 5065 0.22 MB/sec execute 108 sec latency 381.470 ms oleg254-client.virtnet: 1 5139 0.22 MB/sec execute 109 sec latency 87.186 ms oleg254-client.virtnet: 1 5210 0.21 MB/sec execute 110 sec latency 169.934 ms oleg254-client.virtnet: 1 5327 0.21 MB/sec execute 111 sec latency 62.668 ms oleg254-client.virtnet: 1 5453 0.21 MB/sec execute 112 sec latency 125.649 ms oleg254-client.virtnet: 1 5650 0.21 MB/sec execute 113 sec latency 54.350 ms oleg254-client.virtnet: 1 5743 0.21 MB/sec execute 114 sec latency 62.781 ms oleg254-client.virtnet: 1 5905 0.22 MB/sec execute 115 sec latency 77.777 ms oleg254-client.virtnet: 1 5960 0.22 MB/sec execute 116 sec latency 235.915 ms oleg254-client.virtnet: 1 6037 0.22 MB/sec execute 117 sec latency 153.038 ms oleg254-client.virtnet: 1 6314 0.25 MB/sec execute 118 sec latency 164.063 ms oleg254-client.virtnet: 1 6487 0.26 MB/sec execute 119 sec latency 200.608 ms oleg254-client.virtnet: 1 cleanup 120 sec oleg254-client.virtnet: 1 cleanup 121 sec oleg254-client.virtnet: 0 cleanup 122 sec oleg254-client.virtnet: oleg254-client.virtnet: Operation Count AvgLat MaxLat oleg254-client.virtnet: ---------------------------------------- oleg254-client.virtnet: NTCreateX 959 26.885 178.903 oleg254-client.virtnet: Close 735 66.788 43898.033 oleg254-client.virtnet: Rename 44 37.426 99.837 oleg254-client.virtnet: Unlink 164 9.721 39.665 oleg254-client.virtnet: Qpathinfo 942 43.838 34432.969 oleg254-client.virtnet: Qfileinfo 172 0.444 10.805 oleg254-client.virtnet: Qfsinfo 143 0.846 15.572 oleg254-client.virtnet: Sfileinfo 86 19.934 75.035 oleg254-client.virtnet: Find 329 1.469 38.103 oleg254-client.virtnet: WriteX 495 4.716 44.096 oleg254-client.virtnet: ReadX 1527 0.060 20.132 oleg254-client.virtnet: LockX 4 5.680 8.697 oleg254-client.virtnet: UnlockX 4 4.586 7.594 oleg254-client.virtnet: Flush 59 128.744 381.442 oleg254-client.virtnet: oleg254-client.virtnet: Throughput 0.258696 MB/sec 1 clients 1 procs max_latency=43898.053 ms oleg254-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Apr 14 13:07:34 EDT 2026 with return code 0 oleg254-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg254-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg254-client.virtnet: removed directory 'clients/client0' oleg254-client.virtnet: removed directory 'clients' oleg254-client.virtnet: removed 'client.txt' oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: dbench successfully finished PASS 70b (157s) == replay-single test 70c: tar 1mdts recovery ============ 13:07:43 (1776186463) Starting client oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar 134280 tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 7040 2201216 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3727360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 33792 7474176 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets PASS 70c (284s) == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 13:12:27 (1776186747) SKIP: replay-single test_70d needs >= 2 MDTs SKIP 70d (3s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:12:30 (1776186750) SKIP: replay-single test_70e needs >= 2 MDTs SKIP 70e (3s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:12:33 (1776186753) mount clients oleg254-client.virtnet ... Starting client oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 8192 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 12288 7516160 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 8192 3749888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7500800 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Started lustre-OST0000 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... ldlm.namespaces.MGC192.168.202.154@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8b1350791000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8b1350791000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg254-client.virtnet' ... PASS 70f (116s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:14:29 (1776186869) SKIP: replay-single test_71a needs >= 2 MDTs SKIP 71a (3s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:14:32 (1776186872) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (72s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:15:45 (1776186945) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7128 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (55s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:16:41 (1776187001) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (65s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:17:46 (1776187066) SKIP: replay-single test_80a needs >= 2 MDTs SKIP 80a (4s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:17:50 (1776187070) SKIP: replay-single test_80b needs >= 2 MDTs SKIP 80b (4s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:17:55 (1776187075) SKIP: replay-single test_80c needs >= 2 MDTs SKIP 80c (5s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:17:59 (1776187079) SKIP: replay-single test_80d needs >= 2 MDTs SKIP 80d (4s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:18:03 (1776187083) SKIP: replay-single test_80e needs >= 2 MDTs SKIP 80e (3s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:18:06 (1776187086) SKIP: replay-single test_80f needs >= 2 MDTs SKIP 80f (3s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:18:09 (1776187089) SKIP: replay-single test_80g needs >= 2 MDTs SKIP 80g (3s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:18:12 (1776187092) SKIP: replay-single test_80h needs >= 2 MDTs SKIP 80h (4s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:18:16 (1776187096) SKIP: replay-single test_81a needs >= 2 MDTs SKIP 81a (3s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:18:19 (1776187099) SKIP: replay-single test_81b needs >= 2 MDTs SKIP 81b (3s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:18:23 (1776187103) SKIP: replay-single test_81c needs >= 2 MDTs SKIP 81c (3s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:18:26 (1776187106) SKIP: replay-single test_81d needs >= 2 MDTs SKIP 81d (5s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:18:31 (1776187111) SKIP: replay-single test_81e needs >= 2 MDTs SKIP 81e (5s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:18:36 (1776187116) SKIP: replay-single test_81f needs >= 2 MDTs SKIP 81f (4s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:18:40 (1776187120) SKIP: replay-single test_81g needs >= 2 MDTs SKIP 81g (4s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:18:44 (1776187124) SKIP: replay-single test_81h needs >= 2 MDTs SKIP 81h (2s) == replay-single test 84a: stale open during export disconnect ========================================================== 13:18:46 (1776187126) fail_loc=0x80000144 total: 1 open/close in 0.03 seconds: 39.95 ops/second oleg254-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg254-client: oleg254-client: ssh exited with exit code 5 PASS 84a (13s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:18:59 (1776187139) before recovery: unused locks count = 301 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (68s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:20:07 (1776187207) before recovery: unused locks count = 100 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (62s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:21:09 (1776187269) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:) Stopping client oleg254-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) PASS 86 (36s) == replay-single test 87a: write replay ================== 13:21:45 (1776187305) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.359353 s, 23.3 MB/s Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.326787 s, 25.7 MB/s PASS 87a (56s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:22:41 (1776187361) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.435173 s, 19.3 MB/s 8+0 records in 8+0 records out 8 bytes copied, 0.0110158 s, 0.7 kB/s Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes copied, 0.0377866 s, 1.9 kB/s PASS 87b (53s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:23:34 (1776187414) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 15360 7523328 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] R lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 15360 7523328 1% /mnt/lustre before test: last_id = 7201, next_id = 7170 Creating to objid 7201 on ost lustre-OST0000... total: 33 open/close in 0.54 seconds: 60.76 ops/second total: 8 open/close in 0.08 seconds: 99.82 ops/second before recovery: last_id = 7201, next_id = 7170 Stopping /mnt/lustre-mds1 (opts:) on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failover mds1 to oleg254-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg254-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 7233, next_id = 7202 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.117444 s, 4.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.161387 s, 3.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.121173 s, 4.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.10306 s, 5.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.133567 s, 3.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.117206 s, 4.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0823436 s, 6.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0986302 s, 5.3 MB/s -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7170 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7171 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7172 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7173 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7174 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7175 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7176 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7177 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7178 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7179 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7180 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7181 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7182 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7183 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7184 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7185 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7186 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7187 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7188 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7189 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7190 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7191 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7192 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7193 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7194 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7195 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7196 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7197 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7198 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7199 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7200 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7201 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7202 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7203 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7204 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7205 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7206 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7207 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7208 -rw-r--r-- 1 root root 0 Apr 14 13:23 /mnt/lustre/d88.replay-single/f-7209 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7213 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7214 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7215 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7216 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7217 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7218 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7219 -rw-r--r-- 1 root root 524288 Apr 14 13:25 /mnt/lustre/d88.replay-single/f-7220 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0646384 s, 8.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.087905 s, 6.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0946517 s, 5.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0493037 s, 10.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.145964 s, 3.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.107885 s, 4.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.107463 s, 4.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0723307 s, 7.2 MB/s PASS 88 (99s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:25:13 (1776187513) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 1.02026 s, 41.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg254-server Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg254-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete free_before: 7518208 free_after: 7518208 PASS 89 (154s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:27:47 (1776187667) Waiting 90s for '2' Waiting 70s for '2' Waiting 60s for '2' Waiting 50s for '2' Waiting 40s for '2' Waiting 20s for '2' Waiting 10s for '2' Update not seen after 90s: want '2' got '1' replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6470:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:3525:wait_osts_up() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3350:test_90() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6828:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6875:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6701:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:3425:main() Dumping lctl log to /tmp/testlogs//replay-single.test_90.*.1776187762.log rsync: chown "/tmp/testlogs/.replay-single.test_90.debug_log.oleg254-server.1776187762.log.YEXbSJ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_90.dmesg.oleg254-server.1776187762.log.42gJED" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 90 (99s) == replay-single test 93a: replay + reconnect ============ 13:29:32 (1776187772) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0032135 s, 319 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg254-server Stopping /mnt/lustre-ost1 (opts:) on oleg254-server reboot facets: ost1 Failover ost1 to oleg254-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (79s) == replay-single test 93b: replay + reconnect on mds ===== 13:30:51 (1776187851) total: 20 open/close in 0.44 seconds: 45.94 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server reboot facets: mds1 Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (143s) == replay-single test complete, duration 7550 sec ======== 13:33:14 (1776187994) replay-single: FAIL: test_90 wait_update OSTs up on MDT0000 failed