== sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 11:08:22 (1754406502)
#####
The OST-object referenced by the MDT-object should be there;
otherwise, the LFSCK should re-create the missing OST-object.
without '--delay-create-ostobj' option.
#####
Inject failure stub to simulate dangling referenced MDT-object
fail_loc=0x1610
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2881 prealloc_next_id: 2851 count: 31
total: 47 open/close in 0.40 seconds: 117.26 ops/second
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard0': No such file or directory
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288]
DEBUG: prev component: NULL
DEBUG: Component starts at 0, checking if new mirror
DEBUG: Setting first_comp = true (new mirror)
DEBUG: Setting prev = NULL (new mirror)
DEBUG: About to check adjacency, prev=NULL
DEBUG: Checking start <= end: start=0, end=524288
DEBUG: Component validation passed, continuing
DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 2097152]
DEBUG: prev component: found
DEBUG: prev component flags=0x0, extent=[0, 524288]
DEBUG: About to check adjacency, prev=exists
DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 2097152]
DEBUG: prev flags=0x0, curr flags=0x0
DEBUG: Checking start <= end: start=524288, end=2097152
DEBUG: Component validation passed, continuing
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard1': No such file or directory
fail_loc=0
debug=-1
debug_mb=150
debug=-1
debug_mb=150
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2945 prealloc_next_id: 2916 count: 30
total: 30 open/close in 0.48 seconds: 62.10 ops/second
'ls' should fail because of dangling referenced MDT-object
Trigger layout LFSCK to find out dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should fail because of not repair dangling by default
Trigger layout LFSCK to repair dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should success after layout LFSCK repairing
debug_mb=21
debug_mb=21
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy
oleg446-server: oleg446-server.virtnet: executing load_modules_local
oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy
oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy
oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Using TIMEOUT=20