-----============= acceptance-small: replay-single ============----- Tue Sep 1 19:35:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg322-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 59 36 === replay-single: start setup 19:35:24 (1788305724) === oleg322-client.virtnet: executing check_config_client /mnt/lustre oleg322-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg322-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f4cd14d9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f4cd14d9800.idle_timeout=debug disable quota as required oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === replay-single: finish setup 19:35:58 (1788305758) === == replay-single test 0a: empty replay =================== 19:36:00 (1788305760) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:36:09 (1788305769) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:36:36 (1788305796) targets are mounted 19:36:36 (1788305796) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (56s) == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:36:56 (1788305816) Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 19:37:02 (1788305822) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 19:37:25 (1788305845) targets are mounted 19:37:25 (1788305845) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.51 seconds: 39.55 ops/second - unlinked 0 (time 1788305857 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (45s) == replay-single test 0c: check replay-barrier =========== 19:37:41 (1788305861) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:37:51 (1788305871) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:38:17 (1788305897) targets are mounted 19:38:17 (1788305897) facet_failover done Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (105s) == replay-single test 0d: expired recovery with no clients ========================================================== 19:39:26 (1788305966) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:39:35 (1788305975) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:39:57 (1788305997) targets are mounted 19:39:57 (1788305997) facet_failover done Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre PASS 0d (100s) == replay-single test 1: simple create =================== 19:41:06 (1788306066) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:41:15 (1788306075) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:41:38 (1788306098) targets are mounted 19:41:38 (1788306098) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (48s) == replay-single test 2a: touch ========================== 19:41:54 (1788306114) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:42:06 (1788306126) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:42:39 (1788306159) targets are mounted 19:42:39 (1788306159) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (62s) == replay-single test 2b: touch ========================== 19:42:56 (1788306176) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:43:08 (1788306188) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:43:41 (1788306221) targets are mounted 19:43:41 (1788306221) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (61s) == replay-single test 2c: setstripe replay =============== 19:43:57 (1788306237) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:44:08 (1788306248) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:44:30 (1788306270) targets are mounted 19:44:30 (1788306270) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (48s) == replay-single test 2d: setdirstripe replay ============ 19:44:45 (1788306285) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:44:54 (1788306294) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:45:15 (1788306315) targets are mounted 19:45:15 (1788306315) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (46s) == replay-single test 2e: O_CREAT|O_EXCL create replay === 19:45:31 (1788306331) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:45:41 (1788306341) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:46:11 (1788306371) targets are mounted 19:46:11 (1788306371) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (64s) == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 19:46:36 (1788306396) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:46:45 (1788306405) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:47:07 (1788306427) targets are mounted 19:47:07 (1788306427) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (47s) == replay-single test 3b: replay failed open -ENOMEM ===== 19:47:23 (1788306443) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:47:34 (1788306454) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:48:04 (1788306484) targets are mounted 19:48:04 (1788306484) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (55s) == replay-single test 3c: replay failed open -ENOMEM ===== 19:48:18 (1788306498) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:48:28 (1788306508) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:00 (1788306540) targets are mounted 19:49:00 (1788306540) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (55s) == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 19:49:13 (1788306553) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:49:22 (1788306562) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:44 (1788306584) targets are mounted 19:49:44 (1788306584) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (48s) == replay-single test 4b: |x| rm 10 files ================ 19:50:01 (1788306601) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:50:11 (1788306611) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:50:33 (1788306633) targets are mounted 19:50:33 (1788306633) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (49s) == replay-single test 5: |x| 220 open(O_CREAT) =========== 19:50:50 (1788306650) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:51:06 (1788306666) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:51:27 (1788306687) targets are mounted 19:51:27 (1788306687) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (68s) == replay-single test 6a: mkdir + contained create ======= 19:51:58 (1788306718) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:52:07 (1788306727) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:52:40 (1788306760) targets are mounted 19:52:40 (1788306760) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (61s) == replay-single test 6b: |X| rmdir ====================== 19:52:59 (1788306779) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:53:08 (1788306788) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:40 (1788306820) targets are mounted 19:53:40 (1788306820) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (57s) == replay-single test 7: mkdir |X| contained create ====== 19:53:56 (1788306836) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:54:06 (1788306846) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:54:28 (1788306868) targets are mounted 19:54:28 (1788306868) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (47s) == replay-single test 8: creat open |X| close ============ 19:54:43 (1788306883) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:54:52 (1788306892) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:55:24 (1788306924) targets are mounted 19:55:24 (1788306924) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (59s) == replay-single test 9: |X| create (same inum/gen) ====== 19:55:42 (1788306942) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:55:55 (1788306955) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:56:18 (1788306978) targets are mounted 19:56:18 (1788306978) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (56s) == replay-single test 10: create |X| rename unlink ======= 19:56:38 (1788306998) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:56:51 (1788307011) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:57:20 (1788307040) targets are mounted 19:57:20 (1788307040) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (58s) == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 19:57:37 (1788307057) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre new old Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:57:50 (1788307070) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:58:22 (1788307102) targets are mounted 19:58:22 (1788307102) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (68s) == replay-single test 12: open, unlink |X| close ========= 19:58:45 (1788307125) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:58:55 (1788307135) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 19:59:17 (1788307157) targets are mounted 19:59:17 (1788307157) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (47s) == replay-single test 13: open chmod 0 |x| write close === 19:59:32 (1788307172) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 19:59:41 (1788307181) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:00:03 (1788307203) targets are mounted 20:00:03 (1788307203) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (45s) == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:00:18 (1788307218) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:00:26 (1788307226) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:00:48 (1788307248) targets are mounted 20:00:48 (1788307248) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (46s) == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:01:03 (1788307263) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:01:12 (1788307272) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:01:35 (1788307295) targets are mounted 20:01:35 (1788307295) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (47s) == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:01:51 (1788307311) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:02:00 (1788307320) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:02:35 (1788307355) targets are mounted 20:02:35 (1788307355) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (61s) == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:02:53 (1788307373) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:03:04 (1788307384) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:03:29 (1788307409) targets are mounted 20:03:29 (1788307409) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (52s) == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:03:45 (1788307425) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 pid: 50358 will close Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:03:54 (1788307434) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:04:17 (1788307457) targets are mounted 20:04:17 (1788307457) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (49s) == replay-single test 19: mcreate, open, write, rename === 20:04:34 (1788307474) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre old Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:04:45 (1788307485) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:05:17 (1788307517) targets are mounted 20:05:17 (1788307517) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (58s) == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:05:32 (1788307532) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:05:41 (1788307541) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:06:13 (1788307573) targets are mounted 20:06:13 (1788307573) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (57s) == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:06:29 (1788307589) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x240000400 dd: error writing '/mnt/lustre/f20b.replay-single': Cannot send after transport endpoint shutdown pdsh@oleg322-client: oleg322-client: ssh exited with exit code 5 4069+0 records in 4068+0 records out 16662528 bytes (17 MB, 16 MiB) copied, 3.18772 s, 5.2 MB/s Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:06:44 (1788307604) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:07:05 (1788307625) targets are mounted 20:07:05 (1788307625) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg322-server: *.lustre-MDT0000.recovery_status status: COMPLETE sleep 5 for ZFS MDS Waiting for MDT destroys to complete before 4096, after 4096 PASS 20b (69s) == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:07:38 (1788307658) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 -rw-r--r-- 1 root root 1 Sep 1 20:07 /mnt/lustre/f20c.replay-single PASS 20c (11s) == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:07:49 (1788307669) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7533568 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:07:59 (1788307679) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:08:22 (1788307702) targets are mounted 20:08:22 (1788307702) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (48s) == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:08:37 (1788307717) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:08:47 (1788307727) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:19 (1788307759) targets are mounted 20:09:19 (1788307759) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (59s) == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:09:36 (1788307776) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:09:48 (1788307788) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:10:19 (1788307819) targets are mounted 20:10:19 (1788307819) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (59s) == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:10:36 (1788307836) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:10:45 (1788307845) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:08 (1788307868) targets are mounted 20:11:08 (1788307868) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (48s) == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:11:25 (1788307885) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:11:34 (1788307894) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:57 (1788307917) targets are mounted 20:11:57 (1788307917) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (48s) == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:12:12 (1788307932) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:12:22 (1788307942) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:45 (1788307965) targets are mounted 20:12:45 (1788307965) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (48s) == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:13:00 (1788307980) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:13:09 (1788307989) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:31 (1788308011) targets are mounted 20:13:31 (1788308011) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (45s) == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:13:46 (1788308026) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:13:55 (1788308035) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:14:17 (1788308057) targets are mounted 20:14:17 (1788308057) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (47s) == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:14:32 (1788308072) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:14:42 (1788308082) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:15:17 (1788308117) targets are mounted 20:15:17 (1788308117) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (62s) == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:15:34 (1788308134) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2176 2206336 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:15:45 (1788308145) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:16:09 (1788308169) targets are mounted 20:16:09 (1788308169) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (52s) == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:16:26 (1788308186) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2176 2206336 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:16:35 (1788308195) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:17:09 (1788308229) targets are mounted 20:17:09 (1788308229) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (58s) == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:17:24 (1788308244) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 5 PASS 32 (11s) == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:17:35 (1788308255) total: 10 open/close in 0.29 seconds: 34.57 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.33 seconds: 30.59 ops/second PASS 33a (31s) == replay-single test 33b: test fid seq allocation ======= 20:18:07 (1788308287) fail_loc=0x1311 total: 10 open/close in 0.23 seconds: 43.23 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.28 seconds: 36.16 ops/second PASS 33b (31s) == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:18:38 (1788308318) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2176 2206336 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (31s) == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:19:09 (1788308349) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (35s) SKIP: replay-single test_36 skipping ALWAYS excluded test 36 == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:19:46 (1788308386) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2176 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (34s) == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:20:21 (1788308421) - open/close 425 (time 1788308435.56 total 10.02 last 42.41) total: 800 open/close in 19.16 seconds: 41.75 ops/second - unlinked 0 (time 1788308450 ; total 0 ; last 0) total: 400 unlinks in 7 seconds: 57.142857 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2432 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:21:10 (1788308470) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:21:33 (1788308493) targets are mounted 20:21:33 (1788308493) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788308507 ; total 0 ; last 0) total: 400 unlinks in 8 seconds: 50.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (102s) == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:22:03 (1788308523) - open/close 506 (time 1788308536.60 total 10.01 last 50.55) total: 800 open/close in 15.90 seconds: 50.33 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2560 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre - unlinked 0 (time 1788308549 ; total 0 ; last 0) total: 400 unlinks in 6 seconds: 66.666664 unlinks/second Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:22:41 (1788308561) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:23:04 (1788308584) targets are mounted 20:23:04 (1788308584) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788308597 ; total 0 ; last 0) total: 400 unlinks in 8 seconds: 50.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (90s) == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:23:34 (1788308614) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.010481 s, 391 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0322876 s, 127 kB/s PASS 41 (10s) == replay-single test 42: recovery after ost failure ===== 20:23:44 (1788308624) - open/close 475 (time 1788308638.71 total 10.01 last 47.45) total: 800 open/close in 16.64 seconds: 48.08 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2560 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre - unlinked 0 (time 1788308653 ; total 0 ; last 0) total: 400 unlinks in 6 seconds: 66.666664 unlinks/second debug=-1 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 20:24:25 (1788308665) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 20:24:50 (1788308690) targets are mounted 20:24:50 (1788308690) facet_failover done wait for MDS to timeout and recover - unlinked 0 (time 1788308735 ; total 0 ; last 0) total: 400 unlinks in 8 seconds: 50.000000 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (127s) == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:25:51 (1788308751) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2432 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:26:03 (1788308763) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:26:27 (1788308787) targets are mounted 20:26:27 (1788308787) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (64s) == replay-single test 44a: race in target handle connect ========================================================== 20:26:55 (1788308815) at_max=40 1 of 10 (1788308820) service : cur 5 worst 5 (at 1788308786, 35s ago) 4 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 2 of 10 (1788308827) service : cur 5 worst 5 (at 1788308786, 42s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 3 of 10 (1788308835) service : cur 6 worst 6 (at 1788308835, 1s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 4 of 10 (1788308842) service : cur 6 worst 6 (at 1788308835, 9s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 5 of 10 (1788308850) service : cur 6 worst 6 (at 1788308835, 16s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 6 of 10 (1788308857) service : cur 6 worst 6 (at 1788308835, 23s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 7 of 10 (1788308864) service : cur 6 worst 6 (at 1788308835, 30s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 8 of 10 (1788308872) service : cur 6 worst 6 (at 1788308835, 38s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 9 of 10 (1788308879) service : cur 6 worst 6 (at 1788308835, 45s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 10 of 10 (1788308886) service : cur 6 worst 6 (at 1788308835, 52s ago) 6 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (86s) == replay-single test 44b: race in target handle connect ========================================================== 20:28:21 (1788308901) 1 of 10 (1788308902) service : cur 6 worst 6 (at 1788308835, 68s ago) 6 0 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 2 of 10 (1788308924) service : cur 6 worst 6 (at 1788308835, 91s ago) 6 0 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 3 of 10 (1788308948) service : cur 40 worst 40 (at 1788308944, 5s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 4 of 10 (1788308970) service : cur 40 worst 40 (at 1788308944, 28s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 5 of 10 (1788308993) service : cur 40 worst 40 (at 1788308944, 50s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 6 of 10 (1788309015) service : cur 40 worst 40 (at 1788308944, 73s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 7 of 10 (1788309038) service : cur 40 worst 40 (at 1788308944, 95s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 8 of 10 (1788309060) service : cur 40 worst 40 (at 1788308944, 118s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 9 of 10 (1788309083) service : cur 40 worst 40 (at 1788308944, 141s ago) 40 6 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress 10 of 10 (1788309106) service : cur 40 worst 40 (at 1788308944, 163s ago) 1 40 6 0 fail_loc=0x80000704 error: recover: Connection timed out df: /mnt/lustre: Operation already in progress PASS 44b (233s) == replay-single test 44c: race in target handle connect ========================================================== 20:32:14 (1788309134) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 100 create in 1.44 seconds: 69.28 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:32:46 (1788309166) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:33:09 (1788309189) targets are mounted 20:33:09 (1788309189) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (71s) == replay-single test 45: Handle failed close ============ 20:33:25 (1788309205) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 /mnt/lustre/f45.replay-single has type file OK PASS 45 (8s) == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 20:33:33 (1788309213) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:33:57 (1788309237) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:34:20 (1788309260) targets are mounted 20:34:20 (1788309260) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (65s) == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:34:39 (1788309279) total: 20 open/close in 0.67 seconds: 30.01 ops/second Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 20:34:45 (1788309285) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 20:35:12 (1788309312) targets are mounted 20:35:12 (1788309312) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.49 seconds: 40.41 ops/second - unlinked 0 (time 1788309387 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (113s) == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:36:33 (1788309393) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2304 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 20 open/close in 0.49 seconds: 40.60 ops/second Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:36:44 (1788309404) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:37:07 (1788309427) targets are mounted 20:37:07 (1788309427) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.43 seconds: 46.03 ops/second - unlinked 0 (time 1788309490 ; total 0 ; last 0) total: 40 unlinks in 1 seconds: 40.000000 unlinks/second PASS 48 (105s) == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 20:38:18 (1788309498) PASS 50 (14s) == replay-single test 52: time out lock replay (3764) ==== 20:38:32 (1788309512) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7628 fail_loc=0x80000157 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:38:39 (1788309519) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:39:14 (1788309554) targets are mounted 20:39:14 (1788309554) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (64s) == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 20:39:36 (1788309576) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:39:49 (1788309589) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:40:24 (1788309624) targets are mounted 20:40:24 (1788309624) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (65s) == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 20:40:42 (1788309642) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7628 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:40:57 (1788309657) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:41:30 (1788309690) targets are mounted 20:41:30 (1788309690) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (64s) == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 20:41:46 (1788309706) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:41:59 (1788309719) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:42:22 (1788309742) targets are mounted 20:42:22 (1788309742) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (45s) == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 20:42:31 (1788309751) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:42:39 (1788309759) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:43:04 (1788309784) targets are mounted 20:43:04 (1788309784) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (51s) == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 20:43:22 (1788309802) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:43:34 (1788309814) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:44:06 (1788309846) targets are mounted 20:44:06 (1788309846) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (60s) == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 20:44:22 (1788309862) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:44:34 (1788309874) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:45:00 (1788309900) targets are mounted 20:45:00 (1788309900) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (47s) == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 20:45:09 (1788309909) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:45:24 (1788309924) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:45:57 (1788309957) targets are mounted 20:45:57 (1788309957) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (56s) == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 20:46:05 (1788309965) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:46:20 (1788309980) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:46:53 (1788310013) targets are mounted 20:46:53 (1788310013) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (56s) == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 20:47:01 (1788310021) fail_loc=0x8000012b fail_loc=0x0 touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory rm: cannot remove '/mnt/lustre/f55.replay-single': No such file or directory PASS 55 (24s) == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 20:47:25 (1788310045) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2432 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:47:35 (1788310055) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:48:08 (1788310088) targets are mounted 20:48:08 (1788310088) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (75s) == replay-single test 57: test recovery from llog for setattr op ========================================================== 20:48:40 (1788310120) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2432 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:48:51 (1788310131) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:49:25 (1788310165) targets are mounted 20:49:25 (1788310165) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg322-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (75s) == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 20:49:55 (1788310195) fail_loc=0x8000012c - open/close 730 (time 1788310210.13 total 10.01 last 72.95) - open/close 1466 (time 1788310220.13 total 20.01 last 73.55) - open/close 2151 (time 1788310230.13 total 30.02 last 68.48) total: 2500 open/close in 34.47 seconds: 72.54 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:50:44 (1788310244) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:51:07 (1788310267) targets are mounted 20:51:07 (1788310267) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1788310303 ; total 0 ; last 0) total: 2500 unlinks in 26 seconds: 96.153847 unlinks/second PASS 58a (141s) == replay-single test 58b: test replay of setxattr op ==== 20:52:16 (1788310336) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2560 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:52:28 (1788310348) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:52:50 (1788310370) targets are mounted 20:52:50 (1788310370) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) oleg322-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (60s) == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 20:53:16 (1788310396) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 58c (37s) SKIP: replay-single test_59 skipping ALWAYS excluded test 59 == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 20:53:55 (1788310435) total: 200 open/close in 3.14 seconds: 63.61 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2432 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre - unlinked 0 (time 1788310447 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:54:13 (1788310453) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:54:45 (1788310485) targets are mounted 20:54:45 (1788310485) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788310498 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second PASS 60 (70s) == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 20:55:06 (1788310506) - open/close 676 (time 1788310520.63 total 10.00 last 67.59) total: 800 open/close in 11.96 seconds: 66.87 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2688 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre - unlinked 0 (time 1788310531 ; total 0 ; last 0) total: 800 unlinks in 7 seconds: 114.285713 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 20:55:46 (1788310546) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 20:56:11 (1788310571) targets are mounted 20:56:12 (1788310572) facet_failover done Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 20:56:26 (1788310586) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 20:56:51 (1788310611) targets are mounted 20:56:51 (1788310611) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (153s) == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 20:57:40 (1788310660) fail_loc=0x8000013a Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:57:47 (1788310667) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:58:20 (1788310700) targets are mounted 20:58:20 (1788310700) facet_failover done Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 20:58:37 (1788310717) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 20:59:08 (1788310748) targets are mounted 20:59:08 (1788310748) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0333647 s, 123 kB/s PASS 61b (112s) == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 20:59:31 (1788310771) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 20:59:51 (1788310791) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:00:17 (1788310817) targets are mounted 21:00:17 (1788310817) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (65s) == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:00:36 (1788310836) Stopping /mnt/lustre-mds1 (opts:) on oleg322-server fail_loc=0x80000605 Start mgs: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg322-client: oleg322-server: ssh exited with exit code 95 Start of lustre-mdt1/mdt1 on mgs failed 95 fail_loc=0 Start mgs: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (33s) == replay-single test 62: don't mis-drop resent replay === 21:01:09 (1788310869) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2432 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.43 seconds: 57.58 ops/second fail_loc=0x80000707 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:01:20 (1788310880) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:01:41 (1788310901) targets are mounted 21:01:41 (1788310901) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1788310923 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (59s) == replay-single test 65a: AT: verify early replies ====== 21:02:09 (1788310929) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:1.0:1788310967.170364:0:123174:0:(client.c:582:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 34s (25s) req@ffff9f4cc552b800 x1875174271306240/t0(0) o101->lustre-MDT0000-mdc-ffff9f4cc32f1800@192.168.203.122@tcp:12/10 lens 664/66320 e 1 to 0 dl 1788311001 ref 2 fl Rpc:PQr/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 portal 12 : cur 11 worst 40 (at 1788308969, 2005s ago) 36 5 5 5 portal 23 : cur 5 worst 10 (at 1788306153, 4821s ago) 10 5 5 5 portal 30 : cur 5 worst 5 (at 1788306068, 4906s ago) 5 5 0 0 portal 17 : cur 5 worst 5 (at 1788306243, 4731s ago) 5 5 0 5 portal 24 : cur 5 worst 5 (at 1788306596, 4378s ago) 5 5 0 0 portal 13 : cur 10 worst 10 (at 1788307038, 3936s ago) 10 0 0 10 portal 12 : cur 11 worst 40 (at 1788308969, 2014s ago) 36 5 5 5 portal 23 : cur 5 worst 10 (at 1788306153, 4830s ago) 10 5 5 5 portal 30 : cur 5 worst 5 (at 1788306068, 4915s ago) 5 5 0 0 portal 17 : cur 5 worst 5 (at 1788306243, 4740s ago) 5 5 0 5 portal 24 : cur 5 worst 5 (at 1788306596, 4387s ago) 5 5 0 0 portal 13 : cur 10 worst 10 (at 1788307038, 3945s ago) 10 0 0 10 PASS 65a (59s) == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:03:08 (1788310988) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:3.0:1788311025.022208:0:2357:0:(client.c:582:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (26s) req@ffff9f4cd0bf3b80 x1875174271324032/t0(0) o4->lustre-OST0000-osc-ffff9f4cc32f1800@192.168.203.122@tcp:6/4 lens 4584/448 e 1 to 0 dl 1788311060 ref 2 fl Rpc:Qr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 portal 28 : cur 5 worst 5 (at 1788306059, 4973s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1788306061, 4971s ago) 5 5 5 5 portal 17 : cur 5 worst 5 (at 1788306173, 4859s ago) 5 0 0 0 portal 6 : cur 37 worst 37 (at 1788311029, 3s ago) 37 5 0 0 PASS 65b (48s) == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:03:56 (1788311036) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1788308969, 2094s ago) 5 36 5 5 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1788308969, 2103s ago) 5 36 5 5 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 37 worst 40 (at 1788308969, 2115s ago) 37 36 5 5 fail_loc=0 portal 12 : cur 5 worst 40 (at 1788308969, 2127s ago) 37 36 5 5 Current MDT timeout 5, worst 40 PASS 66a (64s) == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:05:00 (1788311100) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (103s) == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:06:43 (1788311203) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (78s) == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 21:08:01 (1788311281) at_history=8 at_history=8 Creating to objid 5441 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.36 seconds: 49.81 ops/second Connected clients: oleg322-client.virtnet oleg322-client.virtnet service : cur 5 worst 5 (at 1788305650, 5665s ago) 1 1 0 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg322-client.virtnet oleg322-client.virtnet service : cur 5 worst 5 (at 1788305650, 5671s ago) 1 1 0 1 0 osc reconnect attempts on 2nd slow PASS 67b (44s) == replay-single test 68: AT: verify slowing locks ======= 21:08:46 (1788311326) at_history=8 at_history=8 fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 PASS 68 (81s) Cleaning up AT ... == replay-single test 70a: check multi client t-f ======== 21:10:07 (1788311407) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (3s) == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 21:10:11 (1788311411) Starting client oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Started clients oleg322-client.virtnet: 192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg322-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=128751 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2432 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 28672 7478272 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:10:32 (1788311432) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-client.virtnet: looking for dbench program oleg322-client.virtnet: /usr/bin/dbench oleg322-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg322-client.virtnet oleg322-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg322-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg322-client.virtnet' oleg322-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg322-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg322-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg322-client.virtnet at Tue Sep 1 21:10:15 EDT 2026 oleg322-client.virtnet: waiting for dbench pid 128787 oleg322-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg322-client.virtnet: oleg322-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg322-client.virtnet: failed to create barrier semaphore oleg322-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg322-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg322-client.virtnet: releasing clients oleg322-client.virtnet: 1 72 3.13 MB/sec warmup 1 sec latency 117.859 ms oleg322-client.virtnet: 1 147 3.13 MB/sec warmup 2 sec latency 127.807 ms oleg322-client.virtnet: 1 214 2.78 MB/sec warmup 3 sec latency 111.382 ms oleg322-client.virtnet: 1 279 2.61 MB/sec warmup 4 sec latency 91.487 ms oleg322-client.virtnet: 1 323 2.34 MB/sec warmup 5 sec latency 129.744 ms oleg322-client.virtnet: 1 381 2.25 MB/sec warmup 6 sec latency 139.133 ms oleg322-client.virtnet: 1 453 2.26 MB/sec warmup 7 sec latency 105.559 ms oleg322-client.virtnet: 1 513 2.21 MB/sec warmup 8 sec latency 104.540 ms oleg322-client.virtnet: 1 639 2.33 MB/sec warmup 9 sec latency 102.367 ms oleg322-client.virtnet: 1 689 2.11 MB/sec warmup 10 sec latency 107.920 ms oleg322-client.virtnet: 1 721 1.93 MB/sec warmup 11 sec latency 210.223 ms oleg322-client.virtnet: 1 743 1.77 MB/sec warmup 12 sec latency 225.580 ms oleg322-client.virtnet: 1 794 1.65 MB/sec warmup 13 sec latency 149.756 ms oleg322-client.virtnet: 1 851 1.54 MB/sec warmup 14 sec latency 116.445 ms oleg322-client.virtnet: 1 870 1.44 MB/sec warmup 15 sec latency 737.283 ms oleg322-client.virtnet: 1 870 1.35 MB/sec warmup 16 sec latency 1739.523 ms oleg322-client.virtnet: 1 870 1.27 MB/sec warmup 17 sec latency 2739.719 ms oleg322-client.virtnet: 1 870 1.20 MB/sec warmup 18 sec latency 3740.022 ms oleg322-client.virtnet: 1 870 1.14 MB/sec warmup 19 sec latency 4740.259 ms oleg322-client.virtnet: 1 870 1.08 MB/sec warmup 20 sec latency 5740.505 ms oleg322-client.virtnet: 1 870 1.03 MB/sec warmup 21 sec latency 6740.894 ms oleg322-client.virtnet: 1 870 0.98 MB/sec warmup 22 sec latency 7741.548 ms oleg322-client.virtnet: 1 870 0.94 MB/sec warmup 23 sec latency 8741.897 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 1 sec latency 10743.616 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 2 sec latency 11744.032 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 3 sec latency 12744.318 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 4 sec latency 13747.975 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 5 sec latency 14748.163 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 6 sec latency 15750.053 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 7 sec latency 16750.229 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 8 sec latency 17750.456 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 9 sec laoleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:10:55 (1788311455) targets are mounted 21:10:55 (1788311455) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2560 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3718144 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7456768 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:11:18 (1788311478) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 tency 18750.702 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 10 sec latency 19750.961 ms oleg322-client.virtnet: 1 870 0.00 MB/sec execute 11 sec latency 20751.614 ms oleg322-client.virtnet: 1 885 0.00 MB/sec execute 12 sec latency 21164.368 ms oleg322-client.virtnet: 1 957 0.01 MB/sec execute 13 sec latency 147.829 ms oleg322-client.virtnet: 1 1028 0.09 MB/sec execute 14 sec latency 120.130 ms oleg322-client.virtnet: 1 1138 0.09 MB/sec execute 15 sec latency 57.326 ms oleg322-client.virtnet: 1 1221 0.09 MB/sec execute 16 sec latency 70.384 ms oleg322-client.virtnet: 1 1396 0.26 MB/sec execute 17 sec latency 74.174 ms oleg322-client.virtnet: 1 1465 0.24 MB/sec execute 18 sec latency 73.220 ms oleg322-client.virtnet: 1 1542 0.23 MB/sec execute 19 sec latency 167.915 ms oleg322-client.virtnet: 1 1621 0.23 MB/sec execute 20 sec latency 67.924 ms oleg322-client.virtnet: 1 1686 0.22 MB/sec execute 21 sec latency 240.692 ms oleg322-client.virtnet: 1 1797 0.21 MB/sec execute 22 sec latency 66.138 ms oleg322-client.virtnet: 1 1884 0.21 MB/sec execute 23 sec latency 71.351 ms oleg322-client.virtnet: 1 1995 0.20 MB/sec execute 24 sec latency 154.885 ms oleg322-client.virtnet: 1 2082 0.20 MB/sec execute 25 sec latency 60.820 ms oleg322-client.virtnet: 1 2198 0.20 MB/sec execute 26 sec latency 62.156 ms oleg322-client.virtnet: 1 2342 0.23 MB/sec execute 27 sec latency 85.433 ms oleg322-client.virtnet: 1 2389 0.23 MB/sec execute 28 sec latency 183.166 ms oleg322-client.virtnet: 1 2445 0.22 MB/sec execute 29 sec latency 217.054 ms oleg322-client.virtnet: 1 2554 0.24 MB/sec execute 30 sec latency 197.327 ms oleg322-client.virtnet: 1 2719 0.33 MB/sec execute 31 sec latency 60.039 ms oleg322-client.virtnet: 1 2909 0.35 MB/sec execute 32 sec latency 105.975 ms oleg322-client.virtnet: 1 3023 0.36 MB/sec execute 33 sec latency 118.595 ms oleg322-client.virtnet: 1 3237 0.38 MB/sec execute 34 sec latency 183.004 ms oleg322-client.virtnet: 1 3365 0.39 MB/sec execute 35 sec latency 96.351 ms oleg322-client.virtnet: 1 3578 0.47 MB/sec execute 36 sec latency 62.647 ms oleg322-client.virtnet: 1 3590 0.45 MB/sec execute 37 sec latency 851.803 ms oleg322-client.virtnet: 1 3590 0.44 MB/sec execute 38 sec latency 1852.592 ms oleg322-client.virtnet: 1 3590 0.43 MB/sec execute 39 sec latency 2854.584 ms oleg322-client.virtnet: 1 3590 0.42 MB/sec execute 40 sec latency 3855.117 ms oleg322-client.virtnet: 1 3590 0.41 MB/sec execute 41 sec latency 4855.453 ms oleg322-client.virtnet: 1 3590 0.40 MB/sec execute 42 sec latency 5855.810 ms oleg322-client.virtnet: 1 3590 0.39 MB/sec execute 43 sec latency 6856.054 ms oleg322-client.virtnet: 1 3590 0.38 MB/sec execute 44 sec latency 7856.585 ms oleg322-client.virtnet: 1 3590 0.37 MB/sec execute 45 sec latency 8857.601 ms oleg322-client.virtnet: 1 3590 0.36 MB/sec execute 46 sec latency 9857.969 ms oleg322-client.virtnet: 1 3590 0.36 MB/sec execute 47 sec latency 10858.236 ms oleg322-client.virtnet: 1 3590 0.35 MB/sec execute 48 sec latency 11858.926 ms oleg322-client.virtnet: 1 3590 0.34 MB/sec execute 49 sec latency 12859.192 ms oleg322-client.virtnet: 1 3590 0.34 MB/sec execute 50 sec latency 13859.435 ms oleg322-client.virtnet: 1 3590 0.33 MB/sec execute 51 sec latency 14860.723 ms oleg322-client.virtnet: 1 3590 0.32 MB/sec execute 52 sec latency 15862.815 ms oleg322-client.virtnet: 1 3590 0.32 MB/sec execute 53 sec latency 16863.028 ms oleg322-clienoleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:11:40 (1788311500) targets are mounted 21:11:40 (1788311500) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2560 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3718144 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7470080 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:12:02 (1788311522) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 t.virtnet: 1 3590 0.31 MB/sec execute 54 sec latency 17864.130 ms oleg322-client.virtnet: 1 3590 0.30 MB/sec execute 55 sec latency 18864.310 ms oleg322-client.virtnet: 1 3590 0.30 MB/sec execute 56 sec latency 19864.459 ms oleg322-client.virtnet: 1 3590 0.29 MB/sec execute 57 sec latency 20866.390 ms oleg322-client.virtnet: 1 3663 0.29 MB/sec execute 58 sec latency 21166.257 ms oleg322-client.virtnet: 1 3767 0.31 MB/sec execute 59 sec latency 67.228 ms oleg322-client.virtnet: 1 3840 0.30 MB/sec execute 60 sec latency 66.956 ms oleg322-client.virtnet: 1 3923 0.30 MB/sec execute 61 sec latency 66.966 ms oleg322-client.virtnet: 1 4005 0.30 MB/sec execute 62 sec latency 71.596 ms oleg322-client.virtnet: 1 4085 0.29 MB/sec execute 63 sec latency 76.992 ms oleg322-client.virtnet: 1 4183 0.29 MB/sec execute 64 sec latency 48.584 ms oleg322-client.virtnet: 1 4257 0.29 MB/sec execute 65 sec latency 161.556 ms oleg322-client.virtnet: 1 4299 0.28 MB/sec execute 66 sec latency 228.362 ms oleg322-client.virtnet: 1 4349 0.28 MB/sec execute 67 sec latency 95.013 ms oleg322-client.virtnet: 1 4377 0.28 MB/sec execute 68 sec latency 178.393 ms oleg322-client.virtnet: 1 4447 0.28 MB/sec execute 69 sec latency 205.391 ms oleg322-client.virtnet: 1 4504 0.27 MB/sec execute 70 sec latency 100.789 ms oleg322-client.virtnet: 1 4597 0.29 MB/sec execute 71 sec latency 144.992 ms oleg322-client.virtnet: 1 4723 0.28 MB/sec execute 72 sec latency 51.639 ms oleg322-client.virtnet: 1 4808 0.28 MB/sec execute 73 sec latency 53.704 ms oleg322-client.virtnet: 1 4969 0.32 MB/sec execute 74 sec latency 63.658 ms oleg322-client.virtnet: 1 5036 0.31 MB/sec execute 75 sec latency 70.438 ms oleg322-client.virtnet: 1 5119 0.31 MB/sec execute 76 sec latency 102.731 ms oleg322-client.virtnet: 1 5197 0.31 MB/sec execute 77 sec latency 88.388 ms oleg322-client.virtnet: 1 5256 0.30 MB/sec execute 78 sec latency 150.171 ms oleg322-client.virtnet: 1 5326 0.30 MB/sec execute 79 sec latency 81.617 ms oleg322-client.virtnet: 1 5385 0.30 MB/sec execute 80 sec latency 82.763 ms oleg322-client.virtnet: 1 5422 0.29 MB/sec execute 81 sec latency 495.142 ms oleg322-client.virtnet: 1 5422 0.29 MB/sec execute 82 sec latency 1496.162 ms oleg322-client.virtnet: 1 5422 0.29 MB/sec execute 83 sec latency 2498.659 ms oleg322-client.virtnet: 1 5422 0.28 MB/sec execute 84 sec latency 3499.117 ms oleg322-client.virtnet: 1 5422 0.28 MB/sec execute 85 sec latency 4499.358 ms oleg322-client.virtnet: 1 5422 0.28 MB/sec execute 86 sec latency 5499.536 ms oleg322-client.virtnet: 1 5422 0.27 MB/sec execute 87 sec latency 6499.742 ms oleg322-client.virtnet: 1 5422 0.27 MB/sec execute 88 sec latency 7501.903 ms oleg322-client.virtnet: 1 5422 0.27 MB/sec execute 89 sec latency 8505.188 ms oleg322-client.virtnet: 1 5422 0.26 MB/sec execute 90 sec latency 9505.631 ms oleg322-client.virtnet: 1 5422 0.26 MB/sec execute 91 sec latency 10506.175 ms oleg322-client.virtnet: 1 5422 0.26 MB/sec execute 92 sec latency 11506.462 ms oleg322-client.virtnet: 1 5422 0.26 MB/sec execute 93 sec latency 12506.779 ms oleg322-client.virtnet: 1 5422 0.25 MB/sec execute 94 sec latency 13506.989 ms oleg322-client.virtnet: 1 5422 0.25 MB/sec execute 95 sec latency 14507.224 ms oleg322-client.virtnet: 1 5422 0.25 MB/sec execute 96 sec latency 15507.798 ms oleg322-client.virtnet: 1 5422 0.25 MB/sec execute 97 sec latency 16508.035 ms oleg322-client.virtnet: 1 5422 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:12:25 (1788311545) targets are mounted 21:12:25 (1788311545) facet_failover done 0.24 MB/sec execute 98 sec latency 17509.650 ms oleg322-client.virtnet: 1 5422 0.24 MB/sec execute 99 sec latency 18510.424 ms oleg322-client.virtnet: 1 5422 0.24 MB/sec execute 100 sec latency 19511.251 ms oleg322-client.virtnet: 1 5422 0.24 MB/sec execute 101 sec latency 20511.389 ms oleg322-client.virtnet: 1 5422 0.23 MB/sec execute 102 sec latency 21512.137 ms oleg322-client.virtnet: 1 5422 0.23 MB/sec execute 103 sec latency 22514.304 ms oleg322-client.virtnet: 1 5422 0.23 MB/sec execute 104 sec latency 23514.692 ms oleg322-client.virtnet: 1 5422 0.23 MB/sec execute 105 sec latency 24514.885 ms oleg322-client.virtnet: 1 5422 0.22 MB/sec execute 106 sec latency 25518.624 ms oleg322-client.virtnet: 1 5422 0.22 MB/sec execute 107 sec latency 26519.020 ms oleg322-client.virtnet: 1 5422 0.22 MB/sec execute 108 sec latency 27519.252 ms oleg322-client.virtnet: 1 5422 0.22 MB/sec execute 109 sec latency 28519.535 ms oleg322-client.virtnet: 1 5422 0.22 MB/sec execute 110 sec latency 29521.098 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 111 sec latency 30521.321 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 112 sec latency 31521.604 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 113 sec latency 32521.814 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 114 sec latency 33522.340 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 115 sec latency 34523.017 ms oleg322-client.virtnet: 1 5422 0.21 MB/sec execute 116 sec latency 35523.276 ms oleg322-client.virtnet: 1 5422 0.20 MB/sec execute 117 sec latency 36523.508 ms oleg322-client.virtnet: 1 5422 0.20 MB/sec execute 118 sec latency 37525.943 ms oleg322-client.virtnet: 1 5422 0.20 MB/sec execute 119 sec latency 38529.831 ms oleg322-client.virtnet: 1 cleanup 120 sec oleg322-client.virtnet: 1 cleanup 121 sec oleg322-client.virtnet: 1 cleanup 122 sec oleg322-client.virtnet: 1 cleanup 123 sec oleg322-client.virtnet: 1 cleanup 124 sec oleg322-client.virtnet: 1 cleanup 125 sec oleg322-client.virtnet: 1 cleanup 126 sec oleg322-client.virtnet: 1 cleanup 127 sec oleg322-client.virtnet: 1 cleanup 128 sec oleg322-client.virtnet: 1 cleanup 129 sec oleg322-client.virtnet: 1 cleanup 130 sec oleg322-client.virtnet: 1 cleanup 131 sec oleg322-client.virtnet: 1 cleanup 132 sec oleg322-client.virtnet: 1 cleanup 133 sec oleg322-client.virtnet: 1 cleanup 134 sec oleg322-client.virtnet: 1 cleanup 135 sec oleg322-client.virtnet: 1 cleanup 136 sec oleg322-client.virtnet: 1 cleanup 137 sec oleg322-client.virtnet: 1 cleanup 138 sec oleg322-client.virtnet: 1 cleanup 139 sec oleg322-client.virtnet: 1 cleanup 140 sec oleg322-client.virtnet: 1 cleanup 141 sec oleg322-client.virtnet: 1 cleanup 142 sec oleg322-client.virtnet: 1 cleanup 143 sec oleg322-client.virtnet: 1 cleanup 144 sec oleg322-client.virtnet: 1 cleanup 145 sec oleg322-client.virtnet: 1 cleanup 146 sec oleg322-client.virtnet: 1 cleanup 147 sec oleg322-client.virtnet: 1 cleanup 148 sec oleg322-client.virtnet: 1 cleanup 149 sec oleg322-client.virtnet: 1 cleanup 150 sec oleg322-client.virtnet: 1 cleanup 151 sec oleg322-client.virtnet: 1 cleanup 152 sec oleg322-client.virtnet: 1 cleanup 153 sec oleg322-client.virtnet: 1 cleanup 154 sec oleg322-client.virtnet: 1 cleanup 155 sec oleg322-client.virtnet: 1 cleanup 156 sec oleg322-client.virtnet: 1 cleanup 157 sec oleg322-client.virtnet: 1 cleanup 158 sec oleg322-client.virtnet: 1 cleanup 159 sec oleg322-client.virtnet: 1 cleanup 160 sec oleg322-client.virtnet: 1 cleanup 161 sec oleg322-client.virtnet: 1 cleanup 162 sec oleg322-client.virtnet: 1 cleanup 163 sec oleg322-client.virtnet: 1 cleanup 164 sec oleg322-client.virtnet: 1 cleanup 165 sec oleg322-client.virtnet: 1 cleanup 166 sec oleg322-client.virtnet: 1 cleanup 167 sec oleg322-client.virtnet: 1 cleanup 168 sec oleg322-client.virtnet: 1 cleanup 169 sec oleg322-client.virtnet: 1 cleanup 170 sec oleg322-client.virtnet: 1 cleanup 171 sec oleg322-client.virtnet: 1 cleanup 172 sec oleg322-client.virtnet: 1 cleanup 173 sec oleg322-client.virtnet: 1 cleanup 174 sec oleg322-client.virtnet: 1 cleanup 175 sec oleg322-client.virtnet: 1 cleanup 176 sec oleg322-client.virtnet: 1 cleanup 177 sec oleg322-client.virtnet: 1 cleanup 178 sec oleg322-client.virtnet: 1 cleanup 179 sec oleg322-client.virtnet: 1 cleanup 180 sec oleg322-client.virtnet: 1 cleanup 181 sec oleg322-client.virtnet: 1 cleanup 182 sec oleg322-client.virtnet: 1 cleanup 183 sec oleg322-client.virtnet: 0 cleanup 184 sec oleg322-client.virtnet: oleg322-client.virtnet: Operation Count AvgLat MaxLat oleg322-client.virtnet: ---------------------------------------- oleg322-client.virtnet: NTCreateX 810 206.291 101727.486 oleg322-client.virtnet: Close 602 6.957 27.770 oleg322-client.virtnet: Rename 35 41.888 79.702 oleg322-client.virtnet: Unlink 142 11.032 34.964 oleg322-client.virtnet: Qpathinfo 745 8.628 162.296 oleg322-client.virtnet: Qfileinfo 128 0.674 10.178 oleg322-client.virtnet: Qfsinfo 120 0.791 19.444 oleg322-client.virtnet: Sfileinfo 73 29.431 81.876 oleg322-client.virtnet: Find 266 2.069 48.556 oleg322-client.virtnet: WriteX 387 4.726 36.920 oleg322-client.virtnet: ReadX 1189 0.325 51.240 oleg322-client.virtnet: LockX 2 9.183 12.302 oleg322-client.virtnet: UnlockX 2 15.660 15.770 oleg322-client.virtnet: Flush 52 113.326 240.674 oleg322-client.virtnet: oleg322-client.virtnet: Throughput 0.200037 MB/sec 1 clients 1 procs max_latency=38529.831 ms oleg322-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg322-client.virtnet at Tue Sep 1 21:13:43 EDT 2026 with return code 0 oleg322-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg322-client.virtnet oleg322-client.virtnet: /mnt/lustre/d70b.replay-single/oleg322-client.virtnet /mnt/lustre/d70b.replay-single/oleg322-client.virtnet oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg322-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg322-client.virtnet: removed directory 'clients/client0' oleg322-client.virtnet: removed directory 'clients' oleg322-client.virtnet: removed 'client.txt' oleg322-client.virtnet: /mnt/lustre/d70b.replay-single/oleg322-client.virtnet oleg322-client.virtnet: dbench successfully finished oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70b (225s) == replay-single test 70c: tar 1mdts recovery ============ 21:13:57 (1788311637) Starting client oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Started clients oleg322-client.virtnet: 192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar 132519 tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4864 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3575808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3535872 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 26624 7111680 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:16:21 (1788311781) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:16:54 (1788311814) targets are mounted 21:16:54 (1788311814) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (242s) == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 21:17:58 (1788311878) SKIP: replay-single test_70d needs >= 2 MDTs SKIP 70d (3s) == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:18:01 (1788311881) SKIP: replay-single test_70e needs >= 2 MDTs SKIP 70e (3s) == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:18:04 (1788311884) mount clients oleg322-client.virtnet ... Starting client oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Started clients oleg322-client.virtnet: 192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 9216 7527424 1% /mnt/lustre ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 21:18:22 (1788311902) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:18:49 (1788311929) targets are mounted 21:18:49 (1788311929) facet_failover done ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... ldlm.namespaces.MGC192.168.203.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f4cc32f1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f4cc32f1800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg322-client.virtnet' ... PASS 70f (70s) == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:19:14 (1788311954) SKIP: replay-single test_71a needs >= 2 MDTs SKIP 71a (3s) == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:19:18 (1788311958) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:19:29 (1788311969) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:19:51 (1788311991) targets are mounted 21:19:51 (1788311991) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (58s) == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:20:17 (1788312017) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7628 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2432 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:20:27 (1788312027) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:21:01 (1788312061) targets are mounted 21:21:01 (1788312061) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (68s) == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:21:25 (1788312085) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:) Stopping client oleg322-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg322-server Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:21:37 (1788312097) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:22:01 (1788312121) targets are mounted 21:22:01 (1788312121) facet_failover done Starting client oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Mount client oleg322-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Started clients oleg322-client.virtnet: 192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (59s) == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:22:24 (1788312144) SKIP: replay-single test_80a needs >= 2 MDTs SKIP 80a (3s) == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:22:27 (1788312147) SKIP: replay-single test_80b needs >= 2 MDTs SKIP 80b (3s) == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:22:30 (1788312150) SKIP: replay-single test_80c needs >= 2 MDTs SKIP 80c (3s) == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:22:34 (1788312154) SKIP: replay-single test_80d needs >= 2 MDTs SKIP 80d (3s) == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:22:37 (1788312157) SKIP: replay-single test_80e needs >= 2 MDTs SKIP 80e (3s) == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:22:40 (1788312160) SKIP: replay-single test_80f needs >= 2 MDTs SKIP 80f (3s) == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:22:43 (1788312163) SKIP: replay-single test_80g needs >= 2 MDTs SKIP 80g (3s) == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:22:47 (1788312167) SKIP: replay-single test_80h needs >= 2 MDTs SKIP 80h (3s) == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:22:50 (1788312170) SKIP: replay-single test_81a needs >= 2 MDTs SKIP 81a (3s) == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:22:53 (1788312173) SKIP: replay-single test_81b needs >= 2 MDTs SKIP 81b (3s) == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:22:56 (1788312176) SKIP: replay-single test_81c needs >= 2 MDTs SKIP 81c (3s) == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 21:22:59 (1788312179) SKIP: replay-single test_81d needs >= 2 MDTs SKIP 81d (4s) == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 21:23:03 (1788312183) SKIP: replay-single test_81e needs >= 2 MDTs SKIP 81e (2s) == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 21:23:06 (1788312186) SKIP: replay-single test_81f needs >= 2 MDTs SKIP 81f (4s) == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 21:23:09 (1788312189) SKIP: replay-single test_81g needs >= 2 MDTs SKIP 81g (2s) == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 21:23:12 (1788312192) SKIP: replay-single test_81h needs >= 2 MDTs SKIP 81h (2s) == replay-single test 84a: stale open during export disconnect ========================================================== 21:23:15 (1788312195) fail_loc=0x80000144 total: 1 open/close in 0.03 seconds: 39.25 ops/second PASS 84a (11s) == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 21:23:25 (1788312205) before recovery: unused locks count = 203 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:23:37 (1788312217) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:23:59 (1788312239) targets are mounted 21:23:59 (1788312239) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (50s) == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 21:24:15 (1788312255) before recovery: unused locks count = 100 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 21:24:34 (1788312274) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:24:57 (1788312297) targets are mounted 21:24:57 (1788312297) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (59s) == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 21:25:14 (1788312314) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:) Stopping client oleg322-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Mount client oleg322-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre Started clients oleg322-client.virtnet: 192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) PASS 86 (26s) == replay-single test 87a: write replay ================== 21:25:40 (1788312340) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.449777 s, 18.7 MB/s Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 21:25:52 (1788312352) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:26:15 (1788312375) targets are mounted 21:26:15 (1788312375) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.255914 s, 32.8 MB/s PASS 87a (52s) == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 21:26:32 (1788312392) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 12288 7526400 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.282086 s, 29.7 MB/s 8+0 records in 8+0 records out 8 bytes copied, 0.0131794 s, 0.6 kB/s Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 21:26:44 (1788312404) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:27:08 (1788312428) targets are mounted 21:27:08 (1788312428) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes copied, 0.0388201 s, 1.9 kB/s PASS 87b (53s) == replay-single test 88: MDS should not assign same objid to different files ========================================================== 21:27:25 (1788312445) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2560 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3758080 1% /mnt/lustre[OST:0] R lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre before test: last_id = 6657, next_id = 6628 Creating to objid 6657 on ost lustre-OST0000... total: 31 open/close in 0.54 seconds: 57.69 ops/second total: 8 open/close in 0.10 seconds: 81.14 ops/second before recovery: last_id = 6657, next_id = 6628 Stopping /mnt/lustre-mds1 (opts:) on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server Failover mds1 to oleg322-server oleg322-server.virtnet Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg322-server oleg322-server.virtnet Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 6689, next_id = 6658 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.169315 s, 3.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0709343 s, 7.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0724239 s, 7.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0530945 s, 9.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0613305 s, 8.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0650128 s, 8.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.144868 s, 3.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.210377 s, 2.5 MB/s -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6628 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6629 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6630 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6631 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6632 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6633 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6634 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6635 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6636 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6637 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6638 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6639 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6640 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6641 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6642 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6643 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6644 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6645 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6646 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6647 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6648 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6649 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6650 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6651 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6652 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6653 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6654 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6655 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6656 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6657 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6658 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6659 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6660 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6661 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6662 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6663 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6664 -rw-r--r-- 1 root root 0 Sep 1 21:27 /mnt/lustre/d88.replay-single/f-6665 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6669 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6670 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6671 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6672 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6673 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6674 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6675 -rw-r--r-- 1 root root 524288 Sep 1 21:29 /mnt/lustre/d88.replay-single/f-6676 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0825125 s, 6.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.105409 s, 5.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0611641 s, 8.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0966802 s, 5.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.142628 s, 3.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0894839 s, 5.9 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.112991 s, 4.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0922512 s, 5.7 MB/s PASS 88 (111s) == replay-single test 89: no disk space leak on late ost connection ========================================================== 21:29:17 (1788312557) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 1.78452 s, 23.5 MB/s -rw-r--r-- 1 root root 41943040 Sep 1 21:29 /mnt/lustre/d89.replay-single/f89.replay-single Stopping /mnt/lustre-ost1 (opts:) on oleg322-server Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:29:43 (1788312583) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:30:19 (1788312619) targets are mounted 21:30:19 (1788312619) facet_failover done Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete free_before: 7520256 free_after: 7520256 PASS 89 (173s) == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 21:32:10 (1788312730) Create the files Fail ost2 lustre-OST0001_UUID, display the list of affected files Stopping /mnt/lustre-ost2 (opts:) on oleg322-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 /mnt/lustre/d90.replay-single/f1 Querying files on shutdown ost2: lfs find --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6562 0x19a2 0x280000400 * /mnt/lustre/d90.replay-single/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6563 0x19a3 0x280000400 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 Failover ost2 to oleg322-server oleg322-server.virtnet Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 PASS 90 (46s) == replay-single test 93a: replay + reconnect ============ 21:32:56 (1788312776) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00718202 s, 143 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 21:33:07 (1788312787) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 21:33:33 (1788312813) targets are mounted 21:33:33 (1788312813) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (85s) == replay-single test 93b: replay + reconnect on mds ===== 21:34:21 (1788312861) total: 20 open/close in 0.55 seconds: 36.10 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 21:34:29 (1788312869) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 21:35:03 (1788312903) targets are mounted 21:35:03 (1788312903) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (143s) == replay-single test complete, duration 7299 sec ======== 21:36:45 (1788313005) === replay-single: start cleanup 21:36:46 (1788313006) === === replay-single: finish cleanup 21:36:58 (1788313018) ===