-----============= acceptance-small: sanity-flr ============----- Sat Jul 18 15:37:16 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg243-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:37:39 (1784403459) === oleg243-client.virtnet: executing check_config_client /mnt/lustre oleg243-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg243-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff937f852d1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff937f852d1000.idle_timeout=debug disable quota as required oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:38:22 (1784403502) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7603] == sanity-flr test 0a: lfs mirror create with -N option == 15:38:25 (1784403505) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:38:44 (1784403524) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:38:48 (1784403528) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:38:52 (1784403532) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:39:20 (1784403560) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:39:23 (1784403563) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:39:27 (1784403567) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff937f852d1000.stats=clear osc.lustre-OST0001-osc-ffff937f852d1000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:39:35 (1784403575) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:39:47 (1784403587) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:39:56 (1784403596) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:40:01 (1784403601) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:40:10 (1784403610) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 15:40:21 (1784403621) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:40:29 (1784403629) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:40:32 (1784403632) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:40:40 (1784403640) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:40:52 (1784403652) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:40:59 (1784403659) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.717319 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189186 s, 5.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:41:10 (1784403670) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.199636 s, 36.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:41:18 (1784403678) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:41:30 (1784403690) Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (144s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:43:54 (1784403834) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.473847 s, 15.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:44:43 (1784403883) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:46:10 (1784403970) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.74168 s, 11.1 MB/s -rw-r--r-- 1 root root 52428800 Jul 18 15:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.234s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.142s user 0m0.002s sys 0m0.169s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.492s user 0m0.002s sys 0m0.187s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff937f852d1000.ost_server_uuid in FULL state after 0 sec PASS 33b (108s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:47:58 (1784404078) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:48:02 (1784404082) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:48:06 (1784404086) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:48:09 (1784404089) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 15:48:18 (1784404098) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.649651 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:48:31 (1784404111) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041274 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702182 s, 14.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:48:39 (1784404119) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:48:47 (1784404127) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012925 s, 792 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00085336 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113907 s, 899 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0026614 s, 385 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:48:58 (1784404138) create test files with size 15746567 .. 8ec13640ad5a984dd71ab08c2c2e2910 - 058a6c80c504d1460215ce78b7d0e9ea - 0cd3096dae3efd09b4a2d5c7ac2e3671 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 15:49:21 (1784404161) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:49:39 (1784404179) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:49:46 (1784404186) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193092 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277422 s, 37.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:49:56 (1784404196) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123011 s, 17.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.328116 s, 12.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275596 s, 7.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 15:50:12 (1784404212) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:50:16 (1784404216) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:50:19 (1784404219) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635724 s, 16.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 15:50:27 (1784404227) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:50:30 (1784404230) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:50:40 (1784404240) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.3047 s, 34.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:50:52 (1784404252) llite.lustre-ffff937f8e7b5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.425886 s, 24.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff937f8dfc6000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:51:05 (1784404265) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:51:44 (1784404304) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:52:24 (1784404344) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:52:29 (1784404349) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184335 s, 17.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228327 s, 13.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8f54f70e (nosync,stale) Mirror 2: 0x8f54f70e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 15c4cb32041e669d7414486e0ba37464 - mirror 1 checksum: d547fed37d785b2e2f9508b86c6f55ca - mirror 2 checksum: 15c4cb32041e669d7414486e0ba37464 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-18T15:52:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-18T15:52:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 15c4cb32041e669d7414486e0ba37464 - mirror 1 checksum: d547fed37d785b2e2f9508b86c6f55ca - mirror 2 checksum: d547fed37d785b2e2f9508b86c6f55ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-18T15:52:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-18T15:52:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:52:42 (1784404362) Failing mds1 on oleg243-server Stopping /mnt/lustre-mds1 (opts:) on oleg243-server 15:52:50 (1784404370) shut down facet: mds1 facet_host: oleg243-server facet_failover_host: oleg243-server Failover mds1 to oleg243-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 15:53:15 (1784404395) targets are mounted 15:53:15 (1784404395) facet_failover done oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521961 s, 20.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:53:35 (1784404415) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:53:38 (1784404418) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:53:42 (1784404422) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00655476 s, 2.5 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:53:50 (1784404430) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:53:54 (1784404434) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:53:57 (1784404437) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:54:20 (1784404460) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:54:49 (1784404489) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:55:12 (1784404512) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:55:36 (1784404536) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 15:55:44 (1784404544) mirror create pid 53956 mirror split pid 53957 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53956 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53957 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 15:56:51 (1784404611) Seed set to 4613 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000016 1784404613.709059 write 0x01ef95 thru 0x02c806 (0xd872 bytes) truncating to largest ever: 0x1df19 000018 1784404613.713186 trunc from 0x02c807 to 0x01df18 (0xe8ee bytes) 000019 1784404613.727563 write 0x02b681 thru 0x035683 (0xa003 bytes) 000021 1784404613.733767 read 0x014a5e thru 0x020bf2 (0xc195 bytes) 000022 1784404613.749434 read 0x0044ad thru 0x00ff01 (0xba55 bytes) 000024 1784404613.767448 mapwrite 0x0033ea thru 0x00e2f0 (0xaf07 bytes) truncating to largest ever: 0x2ecb8 000025 1784404613.934162 trunc from 0x035684 to 0x02ecb7 (0x69cc bytes) 000027 1784404613.986541 write 0x023824 thru 0x02d333 (0x9b10 bytes) 000029 1784404613.988894 read 0x02bba1 thru 0x02ecb7 (0x3117 bytes) 000032 1784404613.995733 mapwrite 0x0317a5 thru 0x03dff0 (0xc84c bytes) 000034 1784404614.115925 mapread 0x022632 thru 0x02fbd1 (0xd5a0 bytes) 000038 1784404614.131662 mapwrite 0x031e48 thru 0x03d7e7 (0xb9a0 bytes) 000040 1784404614.198666 mapwrite 0x019855 thru 0x01e112 (0x48be bytes) 000041 1784404614.257877 trunc from 0x03dff1 to 0x0154ff (0x28af1 bytes) 000042 1784404614.283232 read 0x002235 thru 0x009289 (0x7055 bytes) 000045 1784404614.286717 write 0x01dbd1 thru 0x021ed8 (0x4308 bytes) 000051 1784404614.297176 MIRROR_EXTEND to 2 mirrors 000052 1784404614.774323 mapwrite 0x0371e8 thru 0x03cdc2 (0x5bdb bytes) 000054 1784404615.127507 read 0x003343 thru 0x008ab2 (0x5770 bytes) 000055 1784404615.139900 trunc from 0x03cdc3 to 0x013a14 (0x293ae bytes) 000056 1784404615.169320 read 0x00690f thru 0x009030 (0x2722 bytes) 000057 1784404615.182905 read 0x007d30 thru 0x00b614 (0x38e5 bytes) 000061 1784404615.189919 read 0x004adc thru 0x00867a (0x3b9f bytes) 000063 1784404615.191312 mapwrite 0x0264b8 thru 0x0311f6 (0xad3f bytes) 000064 1784404615.321085 trunc from 0x0311f7 to 0x002f0d (0x2e2e9 bytes) 000065 1784404615.368122 write 0x024bc2 thru 0x02af2d (0x636c bytes) 000066 1784404615.383134 write 0x01d380 thru 0x01e1c3 (0xe44 bytes) 000068 1784404615.397200 trunc from 0x02af2e to 0x0251db (0x5d52 bytes) 000069 1784404615.439682 MIRROR_RESYNC 2 mirrors 000070 1784404616.308767 write 0x017bf2 thru 0x01d4f6 (0x5905 bytes) 000072 1784404616.382402 MIRROR_RESYNC 2 mirrors 000073 1784404616.909247 mapread 0x0181dc thru 0x0251db (0xd000 bytes) 000075 1784404616.940661 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000076 1784404617.071066 read 0x00431e thru 0x0053a6 (0x1089 bytes) 000080 1784404617.080783 mapread 0x001173 thru 0x00d73e (0xc5cc bytes) 000081 1784404617.099834 read 0x00c2ed thru 0x011855 (0x5569 bytes) 000085 1784404617.110654 write 0x01c9ce thru 0x01cb7d (0x1b0 bytes) 000086 1784404617.124759 read 0x004fda thru 0x0106e3 (0xb70a bytes) 000087 1784404617.139778 trunc from 0x0251dc to 0x029b6f (0x4994 bytes) 000090 1784404617.164366 mapread 0x0181a1 thru 0x024e97 (0xccf7 bytes) 000095 1784404617.172871 read 0x008414 thru 0x012d23 (0xa910 bytes) 000097 1784404617.174876 mapread 0x01f9aa thru 0x028c0a (0x9261 bytes) 000099 1784404617.182465 mapwrite 0x018f7b thru 0x021cc9 (0x8d4f bytes) 000100 1784404617.240683 mapwrite 0x01e32b thru 0x026463 (0x8139 bytes) 000104 1784404617.310089 read 0x02529e thru 0x029b6f (0x48d2 bytes) 000107 1784404617.313927 mapread 0x0031d6 thru 0x007cd4 (0x4aff bytes) 000108 1784404617.315680 read 0x01eeb7 thru 0x0213ae (0x24f8 bytes) 000109 1784404617.316536 trunc from 0x029b70 to 0x024d9b (0x4dd4 bytes) 000118 1784404617.350990 mapread 0x020613 thru 0x024d9b (0x4789 bytes) 000119 1784404617.352306 trunc from 0x024d9c to 0x009fd1 (0x1adca bytes) 000120 1784404617.375135 write 0x035fba thru 0x03db19 (0x7b60 bytes) 000121 1784404617.376839 mapread 0x025c48 thru 0x031dab (0xc164 bytes) 000122 1784404617.387719 mapread 0x0398a0 thru 0x03cc94 (0x33f5 bytes) 000123 1784404617.389118 write 0x00048c thru 0x00f912 (0xf487 bytes) 000124 1784404617.391134 mapwrite 0x0293b3 thru 0x02c1f3 (0x2e41 bytes) 000125 1784404617.515529 trunc from 0x03db1a to 0x012755 (0x2b3c4 bytes) 000126 1784404617.534805 mapread 0x007605 thru 0x00dc0b (0x6607 bytes) 000127 1784404617.536090 mapread 0x0106fb thru 0x012755 (0x205b bytes) 000128 1784404617.539635 mapwrite 0x03c4fc thru 0x03dad7 (0x15dc bytes) 000129 1784404617.611521 mapwrite 0x01ca6f thru 0x01d985 (0xf17 bytes) truncating to largest ever: 0x3d8d7 000130 1784404617.655794 trunc from 0x03dad8 to 0x03d8d6 (0x201 bytes) 000131 1784404617.674523 write 0x02c98b thru 0x033d29 (0x739f bytes) 000133 1784404617.688714 mapread 0x011a3a thru 0x01d7c1 (0xbd88 bytes) 000136 1784404617.730753 mapwrite 0x03ff72 thru 0x03ffff (0x8e bytes) 000137 1784404617.798872 read 0x00ba1a thru 0x015f7a (0xa561 bytes) 000139 1784404617.801646 mapwrite 0x01a28a thru 0x028010 (0xdd87 bytes) 000146 1784404617.874088 read 0x014294 thru 0x01a669 (0x63d6 bytes) 000148 1784404617.876590 write 0x037762 thru 0x03984d (0x20ec bytes) 000149 1784404617.878236 mapread 0x016f07 thru 0x018e2d (0x1f27 bytes) 000151 1784404617.882710 mapread 0x03d805 thru 0x03ffff (0x27fb bytes) 000153 1784404617.885677 read 0x0199e7 thru 0x01ea62 (0x507c bytes) 000154 1784404617.888091 mapread 0x00f753 thru 0x018560 (0x8e0e bytes) 000156 1784404617.890757 trunc from 0x040000 to 0x03d753 (0x28ac bytes) 000157 1784404617.906548 mapwrite 0x002237 thru 0x00ea4b (0xc815 bytes) 000158 1784404617.953827 read 0x0386ad thru 0x03b162 (0x2ab6 bytes) 000159 1784404617.954526 mapwrite 0x030a0e thru 0x036b05 (0x60f8 bytes) 000162 1784404617.996755 MIRROR_EXTEND to 2 mirrors 000166 1784404618.621700 MIRROR_EXTEND to 3 mirrors 000167 1784404619.261542 mapwrite 0x03ff03 thru 0x03ffff (0xfd bytes) 000168 1784404619.457828 write 0x03bdc0 thru 0x03ffff (0x4240 bytes) 000169 1784404619.472664 MIRROR_RESYNC 3 mirrors 000170 1784404620.276575 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000173 1784404620.535904 MIRROR_RESYNC 2 mirrors 000174 1784404620.897789 trunc from 0x040000 to 0x0342ba (0xbd45 bytes) 000177 1784404621.002650 trunc from 0x0342bb to 0x027916 (0xc9a4 bytes) 000179 1784404621.026334 trunc from 0x027917 to 0x0000e3 (0x27833 bytes) 000180 1784404621.052824 MIRROR_RESYNC 2 mirrors 000182 1784404621.534134 read 0x0000af thru 0x0000e3 (0x35 bytes) 000184 1784404621.547272 MIRROR_RESYNC 2 mirrors 000185 1784404621.857759 trunc from 0x0000e4 to 0x021c2d (0x21b4a bytes) 000187 1784404621.998942 read 0x01deb9 thru 0x021c2d (0x3d75 bytes) 000190 1784404622.011338 write 0x033d19 thru 0x03ffff (0xc2e7 bytes) 000191 1784404622.015872 read 0x00b7e7 thru 0x0163e9 (0xac03 bytes) 000193 1784404622.026388 mapwrite 0x020b03 thru 0x02134b (0x849 bytes) 000194 1784404622.094767 trunc from 0x040000 to 0x03d657 (0x29a8 bytes) 000195 1784404622.120142 trunc from 0x03d658 to 0x01972b (0x23f2c bytes) 000197 1784404622.138626 mapwrite 0x0024af thru 0x009d2d (0x787f bytes) 000199 1784404622.199646 mapread 0x00a3f6 thru 0x0125ec (0x81f7 bytes) 000202 1784404622.203974 write 0x023884 thru 0x02b81d (0x7f9a bytes) 000203 1784404622.206569 write 0x00ef90 thru 0x01bd42 (0xcdb3 bytes) 000206 1784404622.214881 trunc from 0x02b81e to 0x023ea2 (0x797b bytes) 000207 1784404622.229866 mapwrite 0x00be96 thru 0x00f910 (0x3a7b bytes) 000208 1784404622.312873 write 0x029c4b thru 0x02d16b (0x3521 bytes) 000212 1784404622.318591 read 0x029d83 thru 0x02d16b (0x33e9 bytes) 000213 1784404622.319565 mapread 0x00b472 thru 0x01653f (0xb0ce bytes) 000214 1784404622.326106 trunc from 0x02d16c to 0x022cdf (0xa48c bytes) 000216 1784404622.356371 trunc from 0x022ce0 to 0x00127c (0x21a63 bytes) 000218 1784404622.379683 read 0x000e0f thru 0x00127c (0x46e bytes) 000219 1784404622.381570 write 0x00052b thru 0x0093c8 (0x8e9e bytes) 000220 1784404622.384722 mapread 0x00102b thru 0x0093c8 (0x839e bytes) 000222 1784404622.386730 write 0x03fa6b thru 0x03ffff (0x595 bytes) 000224 1784404622.391101 read 0x022d33 thru 0x02734b (0x4619 bytes) 000226 1784404622.426156 write 0x01d598 thru 0x026281 (0x8cea bytes) 000230 1784404622.437007 write 0x01d86d thru 0x02d3b1 (0xfb45 bytes) 000233 1784404622.447828 trunc from 0x040000 to 0x00cb53 (0x334ac bytes) 000235 1784404622.481156 mapread 0x006b75 thru 0x00cb53 (0x5fdf bytes) 000236 1784404622.482443 read 0x002608 thru 0x00b9cd (0x93c6 bytes) 000237 1784404622.483081 mapread 0x006f2c thru 0x00798d (0xa62 bytes) 000238 1784404622.485573 write 0x01a00d thru 0x01f552 (0x5546 bytes) 000239 1784404622.487839 read 0x0041fd thru 0x004ad7 (0x8db bytes) 000240 1784404622.488783 trunc from 0x01f553 to 0x0356f5 (0x161a3 bytes) 000241 1784404622.515158 read 0x02d530 thru 0x0356f5 (0x81c6 bytes) 000244 1784404622.535862 mapread 0x0137b1 thru 0x021543 (0xdd93 bytes) 000246 1784404622.539553 write 0x01fae0 thru 0x02be9c (0xc3bd bytes) 000247 1784404622.542015 read 0x007b5a thru 0x00db6a (0x6011 bytes) 000250 1784404622.547393 trunc from 0x0356f6 to 0x023cab (0x11a4a bytes) 000251 1784404622.582819 mapwrite 0x00b40e thru 0x01a85c (0xf44f bytes) 000252 1784404622.674085 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000255 1784404622.831094 write 0x0285ca thru 0x0349e4 (0xc41b bytes) 000256 1784404622.832659 read 0x01f10f thru 0x0239e8 (0x48da bytes) 000259 1784404622.841032 mapread 0x01b235 thru 0x026e18 (0xbbe4 bytes) 000261 1784404622.863432 read 0x004339 thru 0x00768f (0x3357 bytes) 000262 1784404622.875461 mapwrite 0x0181f0 thru 0x019523 (0x1334 bytes) 000264 1784404622.916588 trunc from 0x0349e5 to 0x02966f (0xb375 bytes) 000265 1784404622.941896 trunc from 0x029670 to 0x01c4ce (0xd1a1 bytes) 000266 1784404622.969682 trunc from 0x01c4cf to 0x035260 (0x18d92 bytes) 000267 1784404622.997944 trunc from 0x035261 to 0x020c55 (0x1460b bytes) 000269 1784404623.022149 read 0x01ee4b thru 0x020c55 (0x1e0b bytes) 000270 1784404623.032399 trunc from 0x020c56 to 0x00409d (0x1cbb8 bytes) 000271 1784404623.056929 mapwrite 0x02c352 thru 0x032964 (0x6613 bytes) 000276 1784404623.133659 trunc from 0x032965 to 0x002e63 (0x2fb01 bytes) 000280 1784404623.151144 write 0x027b14 thru 0x0307a5 (0x8c92 bytes) 000281 1784404623.153613 read 0x0019d6 thru 0x00d728 (0xbd53 bytes) 000282 1784404623.161316 mapread 0x013cd2 thru 0x01d335 (0x9664 bytes) 000283 1784404623.162577 mapwrite 0x03c882 thru 0x03ffff (0x377e bytes) 000284 1784404623.233185 mapread 0x010178 thru 0x0122ed (0x2176 bytes) 000285 1784404623.234908 trunc from 0x040000 to 0x00570c (0x3a8f3 bytes) 000286 1784404623.249847 read 0x000799 thru 0x00570c (0x4f74 bytes) 000287 1784404623.250582 trunc from 0x00570d to 0x014135 (0xea29 bytes) 000288 1784404623.268071 write 0x013de4 thru 0x01b5db (0x77f8 bytes) 000290 1784404623.282628 mapwrite 0x0321ae thru 0x037a90 (0x58e3 bytes) 000291 1784404623.340063 write 0x002dea thru 0x011fb3 (0xf1ca bytes) 000293 1784404623.343073 mapread 0x015fc1 thru 0x020a7a (0xaaba bytes) 000296 1784404623.349055 trunc from 0x037a91 to 0x0054d8 (0x325b8 bytes) 000299 1784404623.377940 mapread 0x00458b thru 0x0054d8 (0xf4e bytes) 000300 1784404623.382809 trunc from 0x0054d9 to 0x02ed77 (0x2989f bytes) 000305 1784404623.433646 write 0x015410 thru 0x01fdfa (0xa9eb bytes) 000308 1784404623.459301 trunc from 0x02ed78 to 0x01e50f (0x10868 bytes) 000309 1784404623.510582 MIRROR_EXTEND to 2 mirrors 000310 1784404624.231060 trunc from 0x01e510 to 0x023205 (0x4cf6 bytes) 000311 1784404624.352891 write 0x01cdfa thru 0x020567 (0x376e bytes) 000313 1784404624.364664 write 0x00119e thru 0x003cd8 (0x2b3b bytes) 000314 1784404624.373141 trunc from 0x023206 to 0x0340f7 (0x10ef2 bytes) 000315 1784404624.390197 read 0x018209 thru 0x01a8df (0x26d7 bytes) 000316 1784404624.398336 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x340f8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340f8): Mirror 2: 0x72d0965 (stale) CRC-32 checksum value for chunk [0, 0x340f8): Mirror 1: 0x94c50b86 Mirror 3: 0x94c50b86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340f8: skipped 000317 1784404625.012738 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000318 1784404625.326855 write 0x020c79 thru 0x0211e3 (0x56b bytes) 000319 1784404625.406161 read 0x00f703 thru 0x01641d (0x6d1b bytes) 000320 1784404625.415465 write 0x03fea3 thru 0x03ffff (0x15d bytes) 000322 1784404625.417166 trunc from 0x040000 to 0x03a9da (0x5625 bytes) 000324 1784404625.456620 write 0x034ced thru 0x039fe2 (0x52f6 bytes) 000326 1784404625.471577 MIRROR_RESYNC 2 mirrors 000328 1784404625.978059 write 0x0079bb thru 0x00b137 (0x377d bytes) 000331 1784404626.045854 read 0x01cfff thru 0x02a54f (0xd551 bytes) 000332 1784404626.054944 write 0x01c6bb thru 0x029740 (0xd086 bytes) 000334 1784404626.058128 mapread 0x018ee7 thru 0x026cc3 (0xdddd bytes) 000335 1784404626.080097 mapwrite 0x002fa6 thru 0x0093bc (0x6417 bytes) 000336 1784404626.149151 mapwrite 0x026002 thru 0x0294d8 (0x34d7 bytes) 000337 1784404626.193091 mapwrite 0x03cc0f thru 0x03ffff (0x33f1 bytes) 000338 1784404626.283656 mapread 0x00497d thru 0x005771 (0xdf5 bytes) 000339 1784404626.285363 write 0x036063 thru 0x03ce82 (0x6e20 bytes) 000342 1784404626.289431 trunc from 0x040000 to 0x009ea5 (0x3615a bytes) 000345 1784404626.312253 write 0x0215ee thru 0x026225 (0x4c38 bytes) 000347 1784404626.315917 trunc from 0x026226 to 0x0068e5 (0x1f940 bytes) 000348 1784404626.330821 mapwrite 0x022392 thru 0x02a80c (0x847b bytes) 000350 1784404626.429649 mapwrite 0x00eda4 thru 0x0160dc (0x7339 bytes) 000355 1784404626.511431 trunc from 0x02a80d to 0x0086bf (0x2214d bytes) 000362 1784404626.537460 mapwrite 0x02b18b thru 0x032a3d (0x78b3 bytes) 000363 1784404626.615309 read 0x028d59 thru 0x032a3d (0x9ce5 bytes) 000365 1784404626.619288 read 0x02a105 thru 0x02fe05 (0x5d01 bytes) 000367 1784404626.622203 write 0x033c03 thru 0x039d85 (0x6183 bytes) 000371 1784404626.633002 write 0x007cf7 thru 0x008625 (0x92f bytes) 000372 1784404626.635327 trunc from 0x039d86 to 0x003f48 (0x35e3d bytes) 000374 1784404626.652863 trunc from 0x003f49 to 0x010446 (0xc4fe bytes) 000375 1784404626.677640 write 0x001fd8 thru 0x00f028 (0xd051 bytes) 000376 1784404626.689107 mapwrite 0x014160 thru 0x0235e7 (0xf488 bytes) 000378 1784404626.798626 read 0x011441 thru 0x01c9aa (0xb56a bytes) 000379 1784404626.799543 mapread 0x011bb8 thru 0x020a14 (0xee5d bytes) 000381 1784404626.804537 mapread 0x003887 thru 0x009cc2 (0x643c bytes) 000382 1784404626.807703 mapwrite 0x001850 thru 0x00c225 (0xa9d6 bytes) 000383 1784404626.872518 read 0x00d94f thru 0x01366b (0x5d1d bytes) 000386 1784404626.874595 write 0x032abc thru 0x034ad3 (0x2018 bytes) 000387 1784404626.875800 mapread 0x0074df thru 0x00a54f (0x3071 bytes) 000388 1784404626.876965 trunc from 0x034ad4 to 0x00f6a3 (0x25430 bytes) 000389 1784404626.891020 trunc from 0x00f6a4 to 0x031ca3 (0x22600 bytes) 000390 1784404626.906480 mapwrite 0x0116d4 thru 0x01f01d (0xd94a bytes) 000391 1784404626.996050 read 0x00c143 thru 0x013853 (0x7711 bytes) 000394 1784404627.000586 read 0x00fc34 thru 0x012066 (0x2433 bytes) 000395 1784404627.001494 mapread 0x00f767 thru 0x012a41 (0x32db bytes) 000400 1784404627.017053 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000402 1784404627.197935 mapwrite 0x01f68f thru 0x0213c4 (0x1d36 bytes) 000404 1784404627.286063 mapwrite 0x007b8d thru 0x014fd4 (0xd448 bytes) 000408 1784404627.394432 trunc from 0x031ca4 to 0x024e3c (0xce67 bytes) 000409 1784404627.445810 mapwrite 0x03d83d thru 0x03ffff (0x27c3 bytes) 000411 1784404627.580139 mapwrite 0x012ec1 thru 0x01a154 (0x7294 bytes) 000413 1784404627.646667 read 0x013a92 thru 0x023327 (0xf896 bytes) 000414 1784404627.647537 mapwrite 0x016e7a thru 0x01e993 (0x7b1a bytes) 000415 1784404627.712139 MIRROR_EXTEND to 2 mirrors 000416 1784404628.419806 mapwrite 0x01550f thru 0x0191ee (0x3ce0 bytes) 000417 1784404628.596551 read 0x014801 thru 0x0208b2 (0xc0b2 bytes) 000418 1784404628.602826 mapwrite 0x011dbc thru 0x017509 (0x574e bytes) 000420 1784404628.648833 read 0x01d9ce thru 0x020865 (0x2e98 bytes) 000423 1784404628.651524 mapwrite 0x012261 thru 0x01fd6a (0xdb0a bytes) 000427 1784404628.698067 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000428 1784404628.816938 read 0x002897 thru 0x004eac (0x2616 bytes) 000431 1784404628.828096 trunc from 0x040000 to 0x00379e (0x3c861 bytes) 000433 1784404628.847201 write 0x036d86 thru 0x03dbce (0x6e49 bytes) 000434 1784404628.852252 mapwrite 0x00bd3b thru 0x012021 (0x62e7 bytes) 000435 1784404628.945075 trunc from 0x03dbcf to 0x02bb3d (0x12091 bytes) 000437 1784404628.974982 mapread 0x022874 thru 0x02bb3d (0x92ca bytes) 000438 1784404628.990752 read 0x01fad0 thru 0x02985d (0x9d8e bytes) 000441 1784404629.008804 write 0x007d2b thru 0x00a4ca (0x27a0 bytes) 000443 1784404629.015821 write 0x0088e5 thru 0x011d6c (0x9488 bytes) 000444 1784404629.019522 mapread 0x016903 thru 0x02197d (0xb07b bytes) 000445 1784404629.035603 write 0x000a38 thru 0x003471 (0x2a3a bytes) 000447 1784404629.038374 mapread 0x003136 thru 0x007ec4 (0x4d8f bytes) 000450 1784404629.042251 write 0x003b03 thru 0x010b4d (0xd04b bytes) 000454 1784404629.045732 write 0x02ecba thru 0x037ea2 (0x91e9 bytes) 000455 1784404629.049852 read 0x0113e6 thru 0x01815b (0x6d76 bytes) 000457 1784404629.051169 write 0x010fec thru 0x01335b (0x2370 bytes) 000459 1784404629.054806 mapwrite 0x028279 thru 0x0376b9 (0xf441 bytes) 000460 1784404629.162376 mapread 0x002dbb thru 0x005198 (0x23de bytes) 000462 1784404629.165023 mapread 0x00440a thru 0x00e3c0 (0x9fb7 bytes) 000463 1784404629.166342 trunc from 0x037ea3 to 0x0060e4 (0x31dbe bytes) 000464 1784404629.187310 mapread 0x00497d thru 0x005733 (0xdb7 bytes) 000466 1784404629.188984 read 0x001e3b thru 0x0060e4 (0x42aa bytes) 000469 1784404629.191712 trunc from 0x0060e5 to 0x032937 (0x2c853 bytes) 000473 1784404629.208370 mapread 0x000d0a thru 0x005c34 (0x4f2b bytes) 000474 1784404629.209721 write 0x00af82 thru 0x00ebc6 (0x3c45 bytes) 000476 1784404629.219346 write 0x00d1d1 thru 0x018a61 (0xb891 bytes) 000481 1784404629.226950 MIRROR_EXTEND to 2 mirrors 000482 1784404629.832874 read 0x01a215 thru 0x01c5e0 (0x23cc bytes) 000483 1784404629.854739 mapread 0x01d57e thru 0x02891b (0xb39e bytes) 000484 1784404629.901556 mapwrite 0x0187c9 thru 0x026c17 (0xe44f bytes) 000485 1784404630.176348 mapwrite 0x001e35 thru 0x00852b (0x66f7 bytes) 000489 1784404630.285262 trunc from 0x032938 to 0x023b90 (0xeda7 bytes) 000491 1784404630.349875 trunc from 0x023b91 to 0x03c311 (0x18781 bytes) 000492 1784404630.374608 write 0x00d14c thru 0x01950d (0xc3c2 bytes) 000493 1784404630.375839 mapwrite 0x03f227 thru 0x03f5f0 (0x3ca bytes) 000497 1784404630.463982 mapread 0x02ad31 thru 0x02d168 (0x2438 bytes) 000498 1784404630.479552 trunc from 0x03f5f1 to 0x01af05 (0x246eb bytes) 000502 1784404630.501853 trunc from 0x01af06 to 0x02b63c (0x10737 bytes) 000503 1784404630.521606 write 0x01f5be thru 0x024601 (0x5044 bytes) 000504 1784404630.533614 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000505 1784404630.704998 mapwrite 0x01cb7e thru 0x026e16 (0xa299 bytes) 000507 1784404630.766708 read 0x026c86 thru 0x0298d0 (0x2c4b bytes) 000509 1784404630.772576 trunc from 0x02b63d to 0x000c1b (0x2aa21 bytes) 000510 1784404630.785939 mapread 0x0007b2 thru 0x000c1b (0x46a bytes) 000511 1784404630.791775 mapwrite 0x029cea thru 0x02b41c (0x1733 bytes) 000512 1784404630.897403 MIRROR_EXTEND to 2 mirrors 000514 1784404631.759275 trunc from 0x02b41d to 0x006c7c (0x247a0 bytes) 000515 1784404631.841307 write 0x01f780 thru 0x020f7d (0x17fe bytes) 000516 1784404631.850680 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20f7e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20f7e): Mirror 2: 0xfc132e02 (stale) CRC-32 checksum value for chunk [0, 0x20f7e): Mirror 1: 0x2892175a Mirror 3: 0x2892175a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20f7e: skipped 000519 1784404633.030006 write 0x0369be thru 0x0390c9 (0x270c bytes) 000520 1784404633.125680 write 0x01806f thru 0x025164 (0xd0f6 bytes) 000521 1784404633.138000 read 0x012489 thru 0x0209a6 (0xe51e bytes) 000522 1784404633.147915 mapread 0x004739 thru 0x00f3d5 (0xac9d bytes) 000523 1784404633.171897 trunc from 0x0390ca to 0x007f87 (0x31142 bytes) 000524 1784404633.227744 mapread 0x0060cd thru 0x007f87 (0x1ebb bytes) 000529 1784404633.236004 mapwrite 0x030f79 thru 0x037aaa (0x6b32 bytes) 000531 1784404633.414504 read 0x00280f thru 0x005a85 (0x3277 bytes) 000533 1784404633.436797 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37aab): Mirror 3: 0x54fedd18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37aab): Mirror 3: 0x54fedd18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37aab: skipped 000535 1784404633.772072 MIRROR_RESYNC 2 mirrors 000536 1784404634.449191 mapread 0x02d44b thru 0x02fc4d (0x2803 bytes) 000538 1784404634.477902 MIRROR_EXTEND to 3 mirrors 000539 1784404635.091296 mapread 0x01a38b thru 0x01e8f3 (0x4569 bytes) 000541 1784404635.131389 trunc from 0x037aab to 0x003027 (0x34a83 bytes) 000543 1784404635.199006 MIRROR_RESYNC 3 mirrors 000544 1784404635.906429 MIRROR_RESYNC 3 mirrors 000545 1784404636.331016 mapread 0x002f8c thru 0x003027 (0x9c bytes) 000547 1784404636.348519 mapread 0x002836 thru 0x003027 (0x7f2 bytes) 000549 1784404636.349668 mapwrite 0x02d1e3 thru 0x03334a (0x6168 bytes) 000550 1784404636.651683 mapread 0x002076 thru 0x005bd9 (0x3b64 bytes) 000551 1784404636.672907 trunc from 0x03334b to 0x0039fc (0x2f94e bytes) 000552 1784404636.709918 read 0x002105 thru 0x0039fc (0x18f8 bytes) 000554 1784404636.712102 write 0x02adaa thru 0x032734 (0x798b bytes) 000555 1784404636.717663 read 0x016240 thru 0x01f892 (0x9653 bytes) 000556 1784404636.728124 trunc from 0x032735 to 0x02fac4 (0x2c70 bytes) 000558 1784404636.756923 trunc from 0x02fac5 to 0x026082 (0x9a42 bytes) 000559 1784404636.774988 mapread 0x006515 thru 0x011cb7 (0xb7a3 bytes) 000560 1784404636.777746 write 0x002db9 thru 0x00465a (0x18a2 bytes) 000563 1784404636.788784 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26083): Mirror 3: 0x589d2344 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26083): Mirror 3: 0x589d2344 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26083: skipped 000564 1784404637.084430 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000565 1784404637.203889 trunc from 0x026083 to 0x01b74a (0xa938 bytes) 000566 1784404637.219086 read 0x014cf3 thru 0x01651a (0x1828 bytes) 000567 1784404637.225282 mapread 0x01661b thru 0x018aac (0x2492 bytes) 000570 1784404637.237373 write 0x00abfa thru 0x0182d9 (0xd6e0 bytes) 000571 1784404637.243359 mapwrite 0x01d2be thru 0x023838 (0x657b bytes) 000573 1784404637.326647 trunc from 0x023839 to 0x027982 (0x414a bytes) 000575 1784404637.347718 read 0x01bc70 thru 0x01bcc2 (0x53 bytes) 000578 1784404637.351862 trunc from 0x027983 to 0x004d83 (0x22bff bytes) 000579 1784404637.368028 write 0x03d356 thru 0x03ffff (0x2caa bytes) 000580 1784404637.370961 read 0x023990 thru 0x0315ec (0xdc5d bytes) 000581 1784404637.381401 mapread 0x038e70 thru 0x03b97c (0x2b0d bytes) 000583 1784404637.382799 mapread 0x015c1f thru 0x021ffe (0xc3e0 bytes) 000586 1784404637.386772 trunc from 0x040000 to 0x013eb1 (0x2c14e bytes) 000587 1784404637.405348 read 0x00c7dc thru 0x013eb1 (0x76d6 bytes) 000589 1784404637.407306 trunc from 0x013eb2 to 0x016169 (0x22b8 bytes) 000594 1784404637.424903 mapwrite 0x031b5f thru 0x03b120 (0x95c2 bytes) 000595 1784404637.493010 mapwrite 0x005afd thru 0x00df1a (0x841e bytes) 000597 1784404637.534805 read 0x00c92a thru 0x019130 (0xc807 bytes) 000598 1784404637.535581 read 0x0344bd thru 0x03757c (0x30c0 bytes) 000599 1784404637.536176 trunc from 0x03b121 to 0x038b84 (0x259c bytes) 000601 1784404637.557333 read 0x02990d thru 0x0362df (0xc9d3 bytes) 000603 1784404637.560642 mapwrite 0x00e638 thru 0x01c088 (0xda51 bytes) 000606 1784404637.610813 trunc from 0x038b85 to 0x027420 (0x11764 bytes) 000612 1784404637.638326 mapread 0x026e83 thru 0x027420 (0x59e bytes) 000613 1784404637.643907 MIRROR_EXTEND to 2 mirrors 000614 1784404638.264174 mapread 0x010318 thru 0x01a00d (0x9cf6 bytes) 000616 1784404638.298005 write 0x01d12d thru 0x01dc8d (0xb61 bytes) 000622 1784404638.368688 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000624 1784404638.504864 mapread 0x0165f8 thru 0x01de74 (0x787d bytes) 000625 1784404638.531499 mapread 0x01570b thru 0x01606d (0x963 bytes) 000628 1784404638.535325 mapread 0x007947 thru 0x01030e (0x89c8 bytes) 000630 1784404638.545778 mapread 0x01314a thru 0x01a27c (0x7133 bytes) 000631 1784404638.547487 mapwrite 0x0398af thru 0x03ffff (0x6751 bytes) 000632 1784404638.636777 mapread 0x009366 thru 0x012ccc (0x9967 bytes) 000633 1784404638.639751 trunc from 0x040000 to 0x005aad (0x3a552 bytes) 000635 1784404638.663378 write 0x0278b2 thru 0x0338b7 (0xc006 bytes) 000636 1784404638.666864 write 0x028fc9 thru 0x02c148 (0x3180 bytes) 000638 1784404638.670184 write 0x02f349 thru 0x033a1c (0x46d4 bytes) 000639 1784404638.671767 trunc from 0x033a1d to 0x013ae7 (0x1ff35 bytes) 000640 1784404638.686798 read 0x005e45 thru 0x01236f (0xc52b bytes) 000644 1784404638.705395 mapread 0x000db8 thru 0x006090 (0x52d9 bytes) 000645 1784404638.711879 write 0x033b7d thru 0x0345de (0xa62 bytes) 000648 1784404638.715428 trunc from 0x0345df to 0x022d3c (0x118a2 bytes) 000649 1784404638.734262 mapread 0x020d9d thru 0x022d3c (0x1fa0 bytes) 000650 1784404638.750486 trunc from 0x022d3d to 0x016b7a (0xc1c2 bytes) 000652 1784404638.769679 write 0x026d23 thru 0x035abe (0xed9c bytes) 000653 1784404638.773250 read 0x02f016 thru 0x035abe (0x6aa9 bytes) 000654 1784404638.782292 MIRROR_EXTEND to 2 mirrors 000655 1784404639.400545 MIRROR_RESYNC 2 mirrors 000656 1784404639.819951 mapread 0x029ef7 thru 0x02d2e8 (0x33f2 bytes) 000657 1784404639.861063 read 0x0123c9 thru 0x01d47b (0xb0b3 bytes) 000658 1784404639.904726 MIRROR_EXTEND to 3 mirrors 000659 1784404640.679917 write 0x026b9d thru 0x027ce3 (0x1147 bytes) 000660 1784404640.751183 mapwrite 0x02bd2d thru 0x02ce69 (0x113d bytes) 000661 1784404640.794408 mapread 0x014e8e thru 0x02319e (0xe311 bytes) 000662 1784404640.811072 mapwrite 0x01df3a thru 0x02d756 (0xf81d bytes) 000663 1784404640.874541 MIRROR_RESYNC 3 mirrors 000664 1784404641.486448 trunc from 0x035abf to 0x03b4ca (0x5a0c bytes) 000665 1784404641.576650 mapread 0x01be7c thru 0x01f3fb (0x3580 bytes) 000666 1784404641.601487 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b4cb): Mirror 2: 0xabbda965 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4cb): Mirror 2: 0xabbda965 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4cb: skipped 000671 1784404642.099796 MIRROR_RESYNC 2 mirrors 000672 1784404642.676759 read 0x0338b8 thru 0x037d14 (0x445d bytes) 000674 1784404642.684826 mapwrite 0x03b918 thru 0x03ffff (0x46e8 bytes) 000676 1784404642.824954 trunc from 0x040000 to 0x036f3d (0x90c2 bytes) 000678 1784404642.846370 read 0x005467 thru 0x0153d0 (0xff6a bytes) 000679 1784404642.880893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36f3e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f3e): Mirror 2: 0xa0ad4dba (stale) CRC-32 checksum value for chunk [0, 0x36f3e): Mirror 1: 0xa0ad4dba Mirror 3: 0xa0ad4dba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f3e: skipped 000681 1784404643.735650 trunc from 0x036f3e to 0x019c29 (0x1d314 bytes) 000682 1784404643.830667 write 0x007aea thru 0x01385b (0xbd72 bytes) 000683 1784404643.854026 MIRROR_RESYNC 3 mirrors 000684 1784404645.002654 MIRROR_EXTEND to 4 mirrors 000685 1784404645.772466 read 0x018581 thru 0x019c29 (0x16a9 bytes) 000686 1784404645.783787 mapread 0x013881 thru 0x014e9c (0x161c bytes) 000687 1784404645.802971 MIRROR_RESYNC 4 mirrors 000688 1784404646.252782 mapread 0x010f27 thru 0x0132d3 (0x23ad bytes) 000690 1784404646.286613 trunc from 0x019c2a to 0x01a596 (0x96d bytes) 000691 1784404646.421664 MIRROR_RESYNC 4 mirrors 000694 1784404647.635603 write 0x02e9c9 thru 0x02ec82 (0x2ba bytes) 000697 1784404647.705633 read 0x001b55 thru 0x00666e (0x4b1a bytes) 000698 1784404647.717920 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 2: 0x7748474e (stale) Mirror 3: 0x7748474e (stale) Mirror 4: 0x7748474e (stale) CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 1: 0x4abbf016 Mirror 5: 0x4abbf016 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec83: skipped 000699 1784404648.655802 read 0x01ca12 thru 0x01e5e2 (0x1bd1 bytes) 000700 1784404648.716953 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 2: 0x7748474e (stale) Mirror 3: 0x7748474e (stale) Mirror 4: 0x7748474e (stale) CRC-32 checksum value for chunk [0, 0x2ec83): Mirror 1: 0x4abbf016 Mirror 5: 0x4abbf016 Mirror 6: 0x4abbf016 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec83: skipped 000703 1784404649.689799 MIRROR_RESYNC 6 mirrors 000704 1784404650.524874 mapwrite 0x00875c thru 0x00caa1 (0x4346 bytes) 000706 1784404650.717446 mapwrite 0x005556 thru 0x0096ce (0x4179 bytes) 000707 1784404650.769986 trunc from 0x02ec83 to 0x01eabf (0x101c3 bytes) 000708 1784404650.808425 trunc from 0x01eac0 to 0x00342d (0x1b692 bytes) 000709 1784404650.833578 read 0x001e33 thru 0x00342d (0x15fb bytes) 000712 1784404650.843044 trunc from 0x00342e to 0x00f29d (0xbe70 bytes) 000713 1784404650.865023 mapread 0x00e267 thru 0x00f29d (0x1037 bytes) 000715 1784404650.882042 read 0x0077ab thru 0x00a962 (0x31b8 bytes) 000716 1784404650.882749 write 0x03434d thru 0x03ab34 (0x67e8 bytes) 000718 1784404650.892674 mapwrite 0x004c75 thru 0x00a0ef (0x547b bytes) 000720 1784404650.953967 trunc from 0x03ab35 to 0x01be1b (0x1ed19 bytes) 000723 1784404650.997021 MIRROR_RESYNC 6 mirrors 000724 1784404652.221846 write 0x0091e4 thru 0x00b352 (0x216f bytes) 000725 1784404652.315867 mapwrite 0x02dafa thru 0x03c1c8 (0xe6cf bytes) 000726 1784404652.453887 trunc from 0x03c1c9 to 0x023791 (0x18a37 bytes) 000727 1784404652.479345 read 0x0133b2 thru 0x01985e (0x64ad bytes) 000728 1784404652.495799 read 0x0233ba thru 0x023791 (0x3d8 bytes) 000732 1784404652.519733 mapread 0x00deab thru 0x01c89e (0xe9f4 bytes) 000734 1784404652.524830 mapread 0x023363 thru 0x023791 (0x42f bytes) 000736 1784404652.534823 MIRROR_RESYNC 6 mirrors 000737 1784404653.813042 trunc from 0x023792 to 0x0395a9 (0x15e18 bytes) 000738 1784404653.898103 trunc from 0x0395aa to 0x03d562 (0x3fb9 bytes) 000739 1784404653.928417 MIRROR_RESYNC 6 mirrors 000740 1784404654.933287 trunc from 0x03d563 to 0x007706 (0x35e5c bytes) 000746 1784404655.049014 trunc from 0x007707 to 0x011167 (0x9a61 bytes) 000749 1784404655.083336 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11168): Mirror 3: 0xec72da67 (stale) Mirror 4: 0xec72da67 (stale) Mirror 5: 0xec72da67 (stale) Mirror 6: 0xec72da67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11168): Mirror 3: 0xec72da67 (stale) Mirror 4: 0xec72da67 (stale) Mirror 5: 0xec72da67 (stale) Mirror 6: 0xec72da67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11168: skipped 000751 1784404655.461651 mapwrite 0x02e8ab thru 0x032889 (0x3fdf bytes) 000753 1784404655.586705 read 0x01405f thru 0x01ee24 (0xadc6 bytes) 000754 1784404655.641380 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3288a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3288a): Mirror 3: 0x5d01316 (stale) Mirror 4: 0x5d01316 (stale) Mirror 5: 0x5d01316 (stale) Mirror 6: 0x5d01316 (stale) CRC-32 checksum value for chunk [0, 0x3288a): Mirror 1: 0xdaa274ca Mirror 7: 0xdaa274ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3288a: skipped 000755 1784404656.577497 read 0x01eaf2 thru 0x02e1d4 (0xf6e3 bytes) 000757 1784404656.595409 mapwrite 0x038310 thru 0x03be1b (0x3b0c bytes) 000758 1784404656.748000 trunc from 0x03be1c to 0x0050c6 (0x36d55 bytes) 000759 1784404656.774799 mapwrite 0x0206d9 thru 0x02566f (0x4f97 bytes) 000760 1784404656.925396 trunc from 0x025670 to 0x00105f (0x24610 bytes) 000762 1784404656.947485 mapread 0x0008d2 thru 0x00105f (0x78e bytes) 000763 1784404656.954112 mapread 0x000a40 thru 0x00105f (0x620 bytes) 000765 1784404656.955890 trunc from 0x001060 to 0x000395 (0xcca bytes) 000766 1784404656.970963 read 0x00030c thru 0x000395 (0x8a bytes) 000767 1784404656.971699 write 0x0021f8 thru 0x004e02 (0x2c0b bytes) 000769 1784404656.973399 read 0x003b99 thru 0x003dc4 (0x22c bytes) 000770 1784404656.975691 mapwrite 0x032d1c thru 0x03c85d (0x9b42 bytes) 000772 1784404657.144205 mapread 0x032320 thru 0x03c85d (0xa53e bytes) 000775 1784404657.153050 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c85e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c85e): Mirror 3: 0x33806466 (stale) Mirror 4: 0x33806466 (stale) Mirror 5: 0x33806466 (stale) Mirror 6: 0x33806466 (stale) Mirror 7: 0x70409d9e (stale) CRC-32 checksum value for chunk [0, 0x3c85e): Mirror 1: 0x5b2cffd6 Mirror 8: 0x5b2cffd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c85e: skipped 000776 1784404658.032659 read 0x0124e7 thru 0x0223d6 (0xfef0 bytes) 000777 1784404658.052135 MIRROR_RESYNC 7 mirrors 000783 1784404659.103451 write 0x02747d thru 0x02efb4 (0x7b38 bytes) 000784 1784404659.176699 read 0x0109dd thru 0x01a62c (0x9c50 bytes) 000787 1784404659.189883 read 0x020ed6 thru 0x02a841 (0x996c bytes) 000788 1784404659.206801 mapread 0x01ee11 thru 0x02125a (0x244a bytes) 000789 1784404659.208429 mapread 0x00503e thru 0x0140d7 (0xf09a bytes) 000790 1784404659.210619 write 0x0208d3 thru 0x02c221 (0xb94f bytes) 000792 1784404659.213002 trunc from 0x03c85e to 0x00a745 (0x32118 bytes) 000793 1784404659.297846 read 0x006a4c thru 0x00a745 (0x3cfa bytes) 000794 1784404659.298524 mapwrite 0x02adee thru 0x038a53 (0xdc66 bytes) 000795 1784404659.467372 MIRROR_RESYNC 7 mirrors 000796 1784404660.409384 write 0x039faa thru 0x03ffff (0x6056 bytes) 000797 1784404660.466384 MIRROR_RESYNC 7 mirrors 000799 1784404661.684193 write 0x03a135 thru 0x03ffff (0x5ecb bytes) 000800 1784404661.761621 mapread 0x0088ce thru 0x00e164 (0x5897 bytes) 000801 1784404661.780060 mapread 0x019eaf thru 0x020e01 (0x6f53 bytes) 000803 1784404661.795094 mapwrite 0x02c904 thru 0x030b80 (0x427d bytes) 000804 1784404661.870566 read 0x031f1a thru 0x03ef8c (0xd073 bytes) 000807 1784404661.885770 write 0x03d317 thru 0x03ffff (0x2ce9 bytes) 000808 1784404661.887778 mapwrite 0x0183bc thru 0x0207cc (0x8411 bytes) 000809 1784404661.946638 trunc from 0x040000 to 0x02b55b (0x14aa4 bytes) 000810 1784404661.979201 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b55c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b55c): Mirror 3: 0x9fefe8d9 (stale) Mirror 4: 0x9fefe8d9 (stale) Mirror 5: 0x9fefe8d9 (stale) Mirror 6: 0x9fefe8d9 (stale) Mirror 7: 0x9fefe8d9 (stale) Mirror 8: 0x9fefe8d9 (stale) CRC-32 checksum value for chunk [0, 0x2b55c): Mirror 1: 0xe35f2b90 Mirror 9: 0xe35f2b90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b55c: skipped 000811 1784404662.989392 write 0x002ed3 thru 0x012543 (0xf671 bytes) 000812 1784404663.101854 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b55c): Mirror 3: 0x9fefe8d9 (stale) Mirror 4: 0x9fefe8d9 (stale) Mirror 5: 0x9fefe8d9 (stale) Mirror 6: 0x9fefe8d9 (stale) Mirror 8: 0x9fefe8d9 (stale) Mirror 9: 0xe35f2b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b55c): Mirror 3: 0x9fefe8d9 (stale) Mirror 4: 0x9fefe8d9 (stale) Mirror 5: 0x9fefe8d9 (stale) Mirror 6: 0x9fefe8d9 (stale) Mirror 8: 0x9fefe8d9 (stale) Mirror 9: 0xe35f2b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b55c: skipped 000813 1784404663.626496 mapread 0x009492 thru 0x0153c9 (0xbf38 bytes) 000815 1784404663.661641 mapread 0x01af1b thru 0x0273c6 (0xc4ac bytes) 000817 1784404663.680998 mapread 0x01e9e5 thru 0x025da7 (0x73c3 bytes) 000819 1784404663.683731 mapread 0x01dce7 thru 0x02b55b (0xd875 bytes) 000822 1784404663.689018 trunc from 0x02b55c to 0x039e22 (0xe8c7 bytes) 000824 1784404663.709144 write 0x03a515 thru 0x03ffff (0x5aeb bytes) 000826 1784404663.710668 mapwrite 0x02c024 thru 0x034315 (0x82f2 bytes) 000827 1784404663.781782 mapwrite 0x0374c5 thru 0x03ffff (0x8b3b bytes) 000829 1784404663.851167 read 0x03f7da thru 0x03ffff (0x826 bytes) 000830 1784404663.852178 mapwrite 0x01791b thru 0x01d237 (0x591d bytes) 000831 1784404663.899176 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc13f3d90 (stale) Mirror 4: 0xc13f3d90 (stale) Mirror 5: 0xc13f3d90 (stale) Mirror 6: 0xc13f3d90 (stale) Mirror 8: 0xc13f3d90 (stale) Mirror 9: 0x3ba73dbd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18e03a49 Mirror 10: 0x18e03a49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1784404664.749591 trunc from 0x040000 to 0x01cb5b (0x234a4 bytes) 000835 1784404664.828596 write 0x01c102 thru 0x020a19 (0x4918 bytes) 000836 1784404664.850540 read 0x01ed6b thru 0x020a19 (0x1caf bytes) 000837 1784404664.851331 read 0x012459 thru 0x0193ba (0x6f62 bytes) 000838 1784404664.861648 mapwrite 0x036eb5 thru 0x036ec7 (0x13 bytes) 000839 1784404664.997041 mapwrite 0x000f3c thru 0x00f4d8 (0xe59d bytes) 000840 1784404665.082193 read 0x0116f6 thru 0x01b86c (0xa177 bytes) 000841 1784404665.084194 read 0x0295f9 thru 0x030eee (0x78f6 bytes) 000843 1784404665.085552 trunc from 0x036ec8 to 0x03ccd7 (0x5e10 bytes) 000844 1784404665.111399 write 0x020ad2 thru 0x02c005 (0xb534 bytes) 000845 1784404665.115299 read 0x00b199 thru 0x0143e4 (0x924c bytes) 000846 1784404665.117650 mapread 0x01689f thru 0x01c817 (0x5f79 bytes) 000848 1784404665.127617 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ccd8): Mirror 4: 0x816886b3 (stale) Mirror 5: 0x816886b3 (stale) Mirror 6: 0x816886b3 (stale) Mirror 8: 0x816886b3 (stale) Mirror 9: 0xb249fcfa (stale) Mirror 10: 0x1773a491 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ccd8): Mirror 4: 0x816886b3 (stale) Mirror 5: 0x816886b3 (stale) Mirror 6: 0x816886b3 (stale) Mirror 8: 0x816886b3 (stale) Mirror 9: 0xb249fcfa (stale) Mirror 10: 0x1773a491 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ccd8: skipped 000849 1784404665.561079 trunc from 0x03ccd8 to 0x003c07 (0x390d0 bytes) 000851 1784404665.580736 mapwrite 0x016a0f thru 0x0241bc (0xd7ae bytes) 000854 1784404665.672412 mapwrite 0x020ade thru 0x021199 (0x6bc bytes) 000855 1784404665.712476 MIRROR_RESYNC 7 mirrors 000857 1784404666.917384 MIRROR_EXTEND to 8 mirrors 000858 1784404667.556593 mapwrite 0x036a0d thru 0x039eab (0x349f bytes) 000859 1784404667.698746 mapwrite 0x034795 thru 0x03ffff (0xb86b bytes) 000860 1784404667.797638 mapwrite 0x0005ca thru 0x00dc55 (0xd68c bytes) 000861 1784404667.846744 mapwrite 0x036015 thru 0x03c234 (0x6220 bytes) 000863 1784404667.887352 read 0x025fef thru 0x026bcf (0xbe1 bytes) 000864 1784404667.892680 read 0x01e633 thru 0x026fab (0x8979 bytes) 000865 1784404667.905189 mapread 0x02dd30 thru 0x03c888 (0xeb59 bytes) 000868 1784404667.908729 write 0x017854 thru 0x025f58 (0xe705 bytes) 000870 1784404667.912251 write 0x013bd3 thru 0x01af1c (0x734a bytes) 000872 1784404667.914323 write 0x025fce thru 0x02e5c4 (0x85f7 bytes) 000873 1784404667.916409 trunc from 0x040000 to 0x032ec9 (0xd136 bytes) 000874 1784404667.933808 read 0x027b22 thru 0x032ec9 (0xb3a8 bytes) 000876 1784404667.935101 mapwrite 0x0194da thru 0x021822 (0x8349 bytes) 000877 1784404667.995360 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32eca): Mirror 4: 0x8fbac493 (stale) Mirror 5: 0x8fbac493 (stale) Mirror 8: 0x8fbac493 (stale) Mirror 9: 0x8fbac493 (stale) Mirror 10: 0x8fbac493 (stale) Mirror 11: 0x8fbac493 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32eca): Mirror 4: 0x8fbac493 (stale) Mirror 5: 0x8fbac493 (stale) Mirror 8: 0x8fbac493 (stale) Mirror 9: 0x8fbac493 (stale) Mirror 10: 0x8fbac493 (stale) Mirror 11: 0x8fbac493 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32eca: skipped 000879 1784404668.472428 trunc from 0x032eca to 0x0395cb (0x6702 bytes) 000880 1784404668.491142 trunc from 0x0395cc to 0x00c3e2 (0x2d1e9 bytes) 000881 1784404668.510923 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc3e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3e3): Mirror 4: 0x68aa7f38 (stale) Mirror 5: 0x68aa7f38 (stale) Mirror 8: 0x68aa7f38 (stale) Mirror 9: 0x68aa7f38 (stale) Mirror 10: 0x68aa7f38 (stale) Mirror 11: 0x68aa7f38 (stale) CRC-32 checksum value for chunk [0, 0xc3e3): Mirror 1: 0x1106835c Mirror 12: 0x1106835c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3e3: skipped 000882 1784404669.265500 mapwrite 0x036712 thru 0x03ffff (0x98ee bytes) 000884 1784404669.398164 read 0x02556a thru 0x028766 (0x31fd bytes) 000885 1784404669.405868 trunc from 0x040000 to 0x016872 (0x2978d bytes) 000888 1784404669.435070 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x16873): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16873): Mirror 4: 0xb001d92a (stale) Mirror 5: 0xb001d92a (stale) Mirror 8: 0xb001d92a (stale) Mirror 9: 0xb001d92a (stale) Mirror 10: 0xb001d92a (stale) Mirror 11: 0xb001d92a (stale) Mirror 12: 0x70e3a64a (stale) CRC-32 checksum value for chunk [0, 0x16873): Mirror 1: 0x70e3a64a Mirror 13: 0x70e3a64a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16873: skipped 000891 1784404670.380685 mapread 0x014abc thru 0x016872 (0x1db7 bytes) 000892 1784404670.444365 read 0x000edc thru 0x008e4b (0x7f70 bytes) 000893 1784404670.466482 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16873): Mirror 4: 0xb001d92a (stale) Mirror 8: 0xb001d92a (stale) Mirror 9: 0xb001d92a (stale) Mirror 10: 0xb001d92a (stale) Mirror 11: 0xb001d92a (stale) Mirror 12: 0x70e3a64a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16873): Mirror 4: 0xb001d92a (stale) Mirror 8: 0xb001d92a (stale) Mirror 9: 0xb001d92a (stale) Mirror 10: 0xb001d92a (stale) Mirror 11: 0xb001d92a (stale) Mirror 12: 0x70e3a64a (stale) CRC-32 checksum value for chunk [0, 0x16873): Mirror 1: 0x70e3a64a Mirror 13: 0x70e3a64a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16873: skipped 000894 1784404671.146784 write 0x01e323 thru 0x025530 (0x720e bytes) 000895 1784404671.227065 read 0x009258 thru 0x00a75e (0x1507 bytes) 000897 1784404671.237016 read 0x00fd6c thru 0x012280 (0x2515 bytes) 000898 1784404671.249476 read 0x0013b0 thru 0x00ca5b (0xb6ac bytes) 000901 1784404671.255868 MIRROR_RESYNC 8 mirrors 000904 1784404672.729185 write 0x039ec4 thru 0x03ffff (0x613c bytes) 000905 1784404672.865545 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x21832567 (stale) Mirror 8: 0x21832567 (stale) Mirror 9: 0x21832567 (stale) Mirror 10: 0x21832567 (stale) Mirror 11: 0x21832567 (stale) Mirror 12: 0x21832567 (stale) Mirror 13: 0x21832567 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc65c48dd Mirror 14: 0xc65c48dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000907 1784404673.857476 read 0x03395b thru 0x035e17 (0x24bd bytes) 000909 1784404673.896610 MIRROR_RESYNC 9 mirrors 000911 1784404675.057134 mapwrite 0x03e5d7 thru 0x03ffff (0x1a29 bytes) 000915 1784404675.250086 mapwrite 0x02f718 thru 0x03ead8 (0xf3c1 bytes) 000917 1784404675.326155 trunc from 0x040000 to 0x03c2c0 (0x3d3f bytes) 000918 1784404675.363413 mapread 0x0115d6 thru 0x0212a7 (0xfcd2 bytes) 000919 1784404675.409460 trunc from 0x03c2c1 to 0x008aae (0x33812 bytes) 000920 1784404675.424078 mapread 0x0088f7 thru 0x008aae (0x1b8 bytes) 000921 1784404675.431655 write 0x0232e6 thru 0x02de22 (0xab3d bytes) 000922 1784404675.437350 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2de23): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de23): Mirror 4: 0x5d444803 (stale) Mirror 8: 0x5d444803 (stale) Mirror 9: 0x5d444803 (stale) Mirror 10: 0x5d444803 (stale) Mirror 11: 0x5d444803 (stale) Mirror 12: 0x5d444803 (stale) Mirror 13: 0x5d444803 (stale) Mirror 14: 0x5d444803 (stale) CRC-32 checksum value for chunk [0, 0x2de23): Mirror 1: 0xf2dbcad0 Mirror 15: 0xf2dbcad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de23: skipped 000923 1784404676.210578 write 0x02db33 thru 0x02e853 (0xd21 bytes) 000925 1784404676.276187 trunc from 0x02e854 to 0x008211 (0x26642 bytes) 000927 1784404676.302001 read 0x000f54 thru 0x008211 (0x72be bytes) 000929 1784404676.309858 write 0x03fd54 thru 0x03ffff (0x2ac bytes) 000930 1784404676.311024 trunc from 0x040000 to 0x003c66 (0x3c399 bytes) 000931 1784404676.329937 MIRROR_RESYNC 10 mirrors 000932 1784404677.515764 mapread 0x002fce thru 0x003c66 (0xc99 bytes) 000933 1784404677.526826 mapwrite 0x0129b8 thru 0x01c044 (0x968d bytes) 000934 1784404677.716743 mapread 0x01befc thru 0x01c044 (0x149 bytes) 000935 1784404677.719965 write 0x02963c thru 0x0388df (0xf2a4 bytes) 000936 1784404677.722056 mapread 0x02c7a5 thru 0x033899 (0x70f5 bytes) 000938 1784404677.727314 mapwrite 0x014c65 thru 0x01eb51 (0x9eed bytes) 000939 1784404677.799066 trunc from 0x0388e0 to 0x03a64f (0x1d70 bytes) 000941 1784404677.827397 trunc from 0x03a650 to 0x03ac6f (0x620 bytes) 000945 1784404677.857891 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ac70): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac70): Mirror 4: 0xb2acd514 (stale) Mirror 8: 0xb2acd514 (stale) Mirror 9: 0xb2acd514 (stale) Mirror 10: 0xb2acd514 (stale) Mirror 11: 0xb2acd514 (stale) Mirror 12: 0xb2acd514 (stale) Mirror 13: 0xb2acd514 (stale) Mirror 14: 0xb2acd514 (stale) Mirror 15: 0xb2acd514 (stale) CRC-32 checksum value for chunk [0, 0x3ac70): Mirror 1: 0x7a68af5a Mirror 16: 0x7a68af5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac70: skipped 000946 1784404678.900004 trunc from 0x03ac70 to 0x02ba74 (0xf1fb bytes) 000949 1784404679.064912 mapread 0x010347 thru 0x01a5f1 (0xa2ab bytes) 000950 1784404679.090816 mapread 0x011f22 thru 0x01a4ad (0x858c bytes) 000951 1784404679.092000 trunc from 0x02ba75 to 0x03448d (0x8a19 bytes) 000952 1784404679.134278 read 0x00e0bd thru 0x0137b9 (0x56fd bytes) 000953 1784404679.135178 write 0x02ac09 thru 0x03921e (0xe616 bytes) 000954 1784404679.169650 MIRROR_RESYNC 11 mirrors 000956 1784404680.993555 mapread 0x0258b7 thru 0x02e1db (0x8925 bytes) 000957 1784404681.015022 mapwrite 0x00bc94 thru 0x017a3f (0xbdac bytes) 000959 1784404681.206673 mapwrite 0x02fc5c thru 0x03f53a (0xf8df bytes) 000961 1784404681.293514 mapwrite 0x02d9d4 thru 0x030970 (0x2f9d bytes) 000962 1784404681.330233 read 0x025516 thru 0x034668 (0xf153 bytes) 000964 1784404681.336840 mapwrite 0x01a8f4 thru 0x0292dc (0xe9e9 bytes) 000967 1784404681.397419 MIRROR_RESYNC 11 mirrors 000971 1784404683.102967 read 0x03c267 thru 0x03d0fc (0xe96 bytes) 000975 1784404683.115743 MIRROR_EXTEND to 12 mirrors 000976 1784404684.271145 MIRROR_EXTEND to 13 mirrors 000977 1784404685.169750 read 0x01e005 thru 0x021e23 (0x3e1f bytes) 000978 1784404685.179605 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000979 1784404685.585611 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000981 1784404686.109325 MIRROR_RESYNC 11 mirrors 000982 1784404686.577180 mapwrite 0x01581d thru 0x0202d7 (0xaabb bytes) 000984 1784404686.767585 trunc from 0x03f53b to 0x00b67a (0x33ec0 bytes) 000985 1784404686.784480 read 0x001d3b thru 0x00b67a (0x9940 bytes) 000986 1784404686.800474 MIRROR_RESYNC 11 mirrors 000987 1784404688.001888 write 0x02664f thru 0x02e9f8 (0x83aa bytes) 000988 1784404688.114122 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e9f9): Mirror 4: 0xae51b2dd (stale) Mirror 9: 0xae51b2dd (stale) Mirror 10: 0xae51b2dd (stale) Mirror 11: 0xae51b2dd (stale) Mirror 12: 0xae51b2dd (stale) Mirror 13: 0xae51b2dd (stale) Mirror 15: 0xae51b2dd (stale) Mirror 17: 0xae51b2dd (stale) Mirror 18: 0xae51b2dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9f9): Mirror 4: 0xae51b2dd (stale) Mirror 9: 0xae51b2dd (stale) Mirror 10: 0xae51b2dd (stale) Mirror 11: 0xae51b2dd (stale) Mirror 12: 0xae51b2dd (stale) Mirror 13: 0xae51b2dd (stale) Mirror 15: 0xae51b2dd (stale) Mirror 17: 0xae51b2dd (stale) Mirror 18: 0xae51b2dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9f9: skipped 000989 1784404688.596983 read 0x01b67a thru 0x02a481 (0xee08 bytes) 000990 1784404688.608597 write 0x00ae4b thru 0x0152f2 (0xa4a8 bytes) 000991 1784404688.618459 mapread 0x009447 thru 0x00a294 (0xe4e bytes) 000992 1784404688.622392 mapwrite 0x03060e thru 0x030916 (0x309 bytes) 000993 1784404688.708452 trunc from 0x030917 to 0x01dd47 (0x12bcf bytes) 000996 1784404688.722711 read 0x0169d2 thru 0x01dd47 (0x7376 bytes) 000997 1784404688.731839 mapread 0x009c8d thru 0x01418e (0xa502 bytes) 000999 1784404688.740702 write 0x01f330 thru 0x02c864 (0xd535 bytes) 001000 1784404688.751619 trunc from 0x02c865 to 0x01950e (0x13356 bytes) All operations completed A-OK! PASS 70a (83s) == sanity-flr test 71: check mirror extend parent fid ==== 15:58:14 (1784404694) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 15:58:17 (1784404697) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222894 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158634 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 15:58:25 (1784404705) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8194240 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2763744 .. Extending file size to 5677312 .. Extending file size to 8294528 .. Extending file size to 6629696 .. Extending file size to 8468608 .. Extending file size to 6779872 .. Extending file size to 7108768 .. Extending file size to 9177760 .. Extending file size to 8013472 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8101280 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9313920 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9186272 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57108 57109 57112 57113 57114 done 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:59:48 (1784404788) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1986752 .. Extending file size to 4422112 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4921280 .. failed Extending file size to 4610848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4618432 .. failed Extending file size to 6197248 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7972768 .. Extending file size to 7150048 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8316896 .. Extending file size to 8591456 .. Extending file size to 8513216 .. Extending file size to 8757888 .. Extending file size to 8594528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8715232 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58559 58560 58561 58564 58566 done 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:01:07 (1784404867) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120195 s, 17.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354998 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8771c784 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:01:27 (1784404887) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0661597 s, 31.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 16:01:35 (1784404895) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:01:43 (1784404903) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351159 s, 6.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:01:51 (1784404911) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164189 s, 12.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:01:59 (1784404919) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334232 s, 6.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:02:34 (1784404954) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137819 s, 7.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:03:36 (1784405016) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342339 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:04:40 (1784405080) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. Creating new pool test_204f oleg243-server: Pool lustre.test_204f created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340323 s, 6.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg243-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:06:00 (1784405160) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:06:08 (1784405168) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:06:16 (1784405176) oleg243-server: Pool lustre.test_206 created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg243-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:06:41 (1784405201) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645653 s, 16.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796267 s, 13.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe277c8a1 Mirror 3: 0xe277c8a1 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:06:53 (1784405213) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:06:56 (1784405216) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:07:00 (1784405220) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.133489 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1090000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1970000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1970000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1970000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2470000 Read this (4 bytes): BBBB 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:07:37 (1784405257) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0177709 s, 461 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:07:45 (1784405265) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679663 s, 15.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20c lmm_fid: [0x200000404:0x20c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20c lmm_fid: [0x200000404:0x20c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834444 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-18 16:07:46.000000000 -0400 Modify: 2026-07-18 16:07:46.000000000 -0400 Change: 2026-07-18 16:07:48.000000000 -0400 Birth: 2026-07-18 16:07:45.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:07:53 (1784405273) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:07:56 (1784405276) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.551237 s, 74.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0436142 s, 93.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 18 16:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 18 16:07 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:08:04 (1784405284) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:08:11 (1784405291) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1862 sec =========== 16:08:20 (1784405300) === sanity-flr: start cleanup 16:08:21 (1784405301) === === sanity-flr: finish cleanup 16:08:28 (1784405308) ===